What it does
Anthropic describes it as "collecting web content that could potentially contribute to their training."
What blocking it changes
Anthropic says restricting it "signals that the site's future materials should be excluded from our AI model training datasets."
14%
of the 228 sites we've audited with a readable robots.txt block it site-wide.
How to block it
User-agent: ClaudeBot
Disallow: /
How to allow it
User-agent: ClaudeBot
Allow: /
Good to know
Anthropic's page also says it supports Crawl-delay for ClaudeBot.
Sources
What does your robots.txt say?
See how your site treats ClaudeBot and the other major crawlers.
Check my site