Fix guides ยท Crawler access

Let AI search crawlers read your site

Remove the Disallow: / for the AI search crawlers you want answers from. If the block is deliberate, you can ignore this.

Vendor documentation

What we check

We read your robots.txt and work out each crawler's rules. It fails if a search crawler is blocked from the whole site, and warns if one is limited to a few allowed paths. Blocking training crawlers doesn't count against you, and neither does blocking ChatGPT-User, which OpenAI says isn't used to determine what appears in Search.

88%

of 195 sites we could read pass this.

Why it matters

OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," though they can still appear as navigational links.

Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."

Perplexity says PerplexityBot is "not used to crawl content for AI foundation models," and recommends allowing it to appear in its search results.

Search and training are separate decisions. You can block model training and still allow search.

How to fix it

  1. Open /robots.txt and find the groups naming these crawlers, along with any User-agent: * group, which applies to crawlers without their own.
  2. Remove Disallow: / from the crawlers you want to allow, or add an explicit Allow: /.
  3. Check your CDN too. Cloudflare, for example, has separate block settings for search and training crawlers.
  4. OpenAI says robots.txt changes can take about 24 hours to be picked up, and Perplexity says up to 24 hours.
# Allow AI search, block model training
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Fix it yourself with AI

Small fixes shouldn't sit in a queue. Give your AI agent one prompt, and it fixes and deploys for you.

PromptShip is made by the team behind this checker.

Sources

Fixed it?

Run your site through the checker, and re-check after you ship to see what improved.

Check my site

Related guides