AI crawlers
What each crawler does, what blocking it changes, and the exact robots.txt rules, from the vendors' own documentation.
| Crawler | Operator | Used for | Blocked by |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search | 7% of sites |
| ChatGPT-User | OpenAI | ChatGPT | 7% of sites |
| GPTBot | OpenAI | Model training | 13% of sites |
| Claude-SearchBot | Anthropic | Claude search | 8% of sites |
| Claude-User | Anthropic | Claude | 7% of sites |
| ClaudeBot | Anthropic | Model training | 14% of sites |
| PerplexityBot | Perplexity | Perplexity search | 11% of sites |
| Googlebot | Google Search | 0% of sites | |
| Google-Extended | Gemini training and grounding | 12% of sites | |
| Bingbot | Microsoft | Bing search | 1% of sites |
| Applebot-Extended | Apple | Apple AI training | 14% of sites |
| CCBot | Common Crawl | Open web archive | 16% of sites |
Share of the 228 sites we've audited with a readable robots.txt that block the crawler site-wide.