Robotst.txt are a wonderful tool that you can use to instruct search engine crawlers how you would want it to crawl your website. They are used mainly to avoid overloading your site with requests, although it isn’t a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
Despite it robots.txt being a useful tool, it isn’t the most powerful, but at least you know that it can be used to prevent your site from being skipped over by Google’s crawler.
Dan Taylor shows us the most common problems with robots.txt files, as well as the impact they might have on your website and it’s presence on Google’s search engine. In his column, Dan gives us the information we need to fix issues if you think this has happened.
