info What is Robots.txt?

It's a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. It's part of the Robots Exclusion Protocol.