robots

Suppose You want to Disallow two directories at your website i.e. some-directory and another-directory located at: http://www.yourwebsite.com/some-directory http://www.yourwebsite.com/another-directory to be crawled by search engines and Allow all other directories and page of whole website to be indexed and crawled then here is how to do it in a right way.

A robots.txt file restricts or allow Search Engine Robots (known as “bots”) that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents or allow them from accessing certain pages. If you do not put a robots.txt file in […]