What it does
Anthropic says it "navigates the web to improve search result quality for users."
What blocking it changes
Anthropic says disabling it prevents indexing your content for search optimization, "which may reduce your site's visibility and accuracy in user search results."
8%
of the 228 sites we've audited with a readable robots.txt block it site-wide.
How to block it
User-agent: Claude-SearchBot
Disallow: /
How to allow it
User-agent: Claude-SearchBot
Allow: /
Good to know
Anthropic says its bots honor robots.txt.
Sources
What does your robots.txt say?
See how your site treats Claude-SearchBot and the other major crawlers.
Check my site