The Past, Present, And Future Of Robots.txt

A robots.txt file allows search engine crawlers which ULRs the crawler can access on a website. Basically, this happens so that your site doesn’t get overloaded with requests. What robots.txt doesn’t do is keep a web page out of Google. In order to that, you would block indexing with noindex or password-protect them.

In this Google Search Central episode of Search Off the Record, we find Gary, Lizzi and Martin from the Google Search team discuss robots.txt with Product Counsel for Google, David Price.

They talk about human initiated crawl requests versus bot initiated crawl requests, the robots.txt file along with more!

Scott Davenport

Scott Davenport is the content writer and social media man of Thrive Business Marketing and Thrive HVAC in Portland Oregon. Writing about the current events of the SEO world, as well as tips and advice that fellow SEOs could use to improve their own SEO campaigns and shares it for the whole world to see!

Leave a Comment

Your email address will not be published. Required fields are marked *

Ready To Thrive?

Get in touch with us today to start growing your business, or if you’d like to discuss campaign options.