Robots.txt guide
Basic tips on how search engines read your robots.txt file.
What is robots.txt?
It is a simple text file at the root of your site (for example https://example.com/robots.txt) that tells search engine crawlers which areas they can or cannot crawl.
Common examples
User-agent: *
Disallow: /admin/
Allow: /This allows all crawlers to access the whole site except the /admin/ area.