What we check
We read your robots.txt and work out each crawler's rules. It fails if a search crawler is blocked from the whole site, and warns if one is limited to a few allowed paths. Blocking training crawlers doesn't count against you, and neither does blocking ChatGPT-User, which OpenAI says isn't used to determine what appears in Search.
of 195 sites we could read pass this.
Why it matters
OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," though they can still appear as navigational links.
Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."
Perplexity says PerplexityBot is "not used to crawl content for AI foundation models," and recommends allowing it to appear in its search results.
Search and training are separate decisions. You can block model training and still allow search.
How to fix it
- Open
/robots.txtand find the groups naming these crawlers, along with anyUser-agent: *group, which applies to crawlers without their own. - Remove
Disallow: /from the crawlers you want to allow, or add an explicitAllow: /. - Check your CDN too. Cloudflare, for example, has separate block settings for search and training crawlers.
- OpenAI says robots.txt changes can take about 24 hours to be picked up, and Perplexity says up to 24 hours.
# Allow AI search, block model training
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Fix it yourself with AI
Small fixes shouldn't sit in a queue. Give your AI agent one prompt, and it fixes and deploys for you.
PromptShip is made by the team behind this checker.
Sources
Fixed it?
Run your site through the checker, and re-check after you ship to see what improved.
Check my site