AI crawlers ยท Anthropic

ClaudeBot

Used for: Model training.

What it does

Anthropic describes it as "collecting web content that could potentially contribute to their training."

What blocking it changes

Anthropic says restricting it "signals that the site's future materials should be excluded from our AI model training datasets."

14%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: ClaudeBot
Disallow: /

How to allow it

User-agent: ClaudeBot
Allow: /

Good to know

Anthropic's page also says it supports Crawl-delay for ClaudeBot.

Sources

What does your robots.txt say?

See how your site treats ClaudeBot and the other major crawlers.

Check my site

Other crawlers