What we check
We fetch /robots.txt and warn if it's missing or returns an HTML page instead of the file.
of 198 sites we could read pass this.
Why it matters
Google says robots.txt "tells search engine crawlers which URLs the crawler can access on your site." It's a published standard, RFC 9309.
A missing file isn't a block, which is why this only warns. Under RFC 9309 a missing robots.txt means no restrictions.
Google also warns it "is not a mechanism for keeping a web page out of Google." Use noindex or password protection for that.
How to fix it
- Create a plain-text file at
/robots.txt. - Add your rules and a
Sitemap:line. - Check that it returns plain text, not your site's normal HTML page.
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Fix it yourself with AI
Small fixes shouldn't sit in a queue. Give your AI agent one prompt, and it fixes and deploys for you.
PromptShip is made by the team behind this checker.
Sources
Fixed it?
Run your site through the checker, and re-check after you ship to see what improved.
Check my site