Nofollow is an attribute used in HTML to tell the search engine to follow the links or not, to give credit to links or not. It stops crawler if you have used this attribute at a link and crawler won’t escape from that link and it won’t give page rank to that link you have [...]
Suppose You want to Disallow two directories at your website i.e. some-directory and another-directory located at: http://www.yourwebsite.com/some-directory http://www.yourwebsite.com/another-directory to be crawled by search engines and Allow all other directories and page of whole website to be indexed and crawled then here is how to do it in a right way.
This online free sitemap Generator allows you generate XML Sitemaps, Yahoo and Google Sitemaps, check your site for errors and much more! You just enter website, choose options and hit start button. After it is done you can export your website data in many formats as: URL list in txt file CSV File XML File [...]
A robots.txt file restricts or allow Search Engine Robots (known as “bots”) that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents or allow them from accessing certain pages. If you do not put a robots.txt file in [...]