Fix guides ยท Crawler access

Add a robots.txt file

Add a robots.txt file at your site root stating which crawlers may access what.

Published standard

What we check

We fetch /robots.txt and warn if it's missing or returns an HTML page instead of the file.

91%

of 198 sites we could read pass this.

Why it matters

Google says robots.txt "tells search engine crawlers which URLs the crawler can access on your site." It's a published standard, RFC 9309.

A missing file isn't a block, which is why this only warns. Under RFC 9309 a missing robots.txt means no restrictions.

Google also warns it "is not a mechanism for keeping a web page out of Google." Use noindex or password protection for that.

How to fix it

  1. Create a plain-text file at /robots.txt.
  2. Add your rules and a Sitemap: line.
  3. Check that it returns plain text, not your site's normal HTML page.
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Fix it yourself with AI

Small fixes shouldn't sit in a queue. Give your AI agent one prompt, and it fixes and deploys for you.

PromptShip is made by the team behind this checker.

Sources

Fixed it?

Run your site through the checker, and re-check after you ship to see what improved.

Check my site

Related guides