Robots.txt Can be a file that exists on the foundation directory of each Internet site and can be utilized to instruct search engines like google and yahoo on which directories/documents of the website they're able to crawl and contain in their index. As outlined by Rubright, an improved technique is https://news.rafeeg.ae