What it does
Anthropic says "when individuals ask questions to Claude, it may access websites using a Claude-User agent."
What blocking it changes
Anthropic says disabling it prevents retrieving your content in response to a user query, "which may reduce your site's visibility for user-directed web search."
7%
of the 228 sites we've audited with a readable robots.txt block it site-wide.
How to block it
User-agent: Claude-User
Disallow: /
How to allow it
User-agent: Claude-User
Allow: /
Good to know
Anthropic says its bots honor robots.txt.
Sources
What does your robots.txt say?
See how your site treats Claude-User and the other major crawlers.
Check my site