AI crawlers ยท Anthropic

Claude-SearchBot

Used for: Claude search.

What it does

Anthropic says it "navigates the web to improve search result quality for users."

What blocking it changes

Anthropic says disabling it prevents indexing your content for search optimization, "which may reduce your site's visibility and accuracy in user search results."

8%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: Claude-SearchBot
Disallow: /

How to allow it

User-agent: Claude-SearchBot
Allow: /

Good to know

Anthropic says its bots honor robots.txt.

Sources

What does your robots.txt say?

See how your site treats Claude-SearchBot and the other major crawlers.

Check my site

Other crawlers