AI crawlers

What each crawler does, what blocking it changes, and the exact robots.txt rules, from the vendors' own documentation.

CrawlerOperatorUsed forBlocked by
OAI-SearchBot OpenAI ChatGPT search 7% of sites
ChatGPT-User OpenAI ChatGPT 7% of sites
GPTBot OpenAI Model training 13% of sites
Claude-SearchBot Anthropic Claude search 8% of sites
Claude-User Anthropic Claude 7% of sites
ClaudeBot Anthropic Model training 14% of sites
PerplexityBot Perplexity Perplexity search 11% of sites
Googlebot Google Google Search 0% of sites
Google-Extended Google Gemini training and grounding 12% of sites
Bingbot Microsoft Bing search 1% of sites
Applebot-Extended Apple Apple AI training 14% of sites
CCBot Common Crawl Open web archive 16% of sites

Share of the 228 sites we've audited with a readable robots.txt that block the crawler site-wide.

See how your site treats them →