AI crawlers ยท Anthropic

Claude-User

Used for: Claude.

What it does

Anthropic says "when individuals ask questions to Claude, it may access websites using a Claude-User agent."

What blocking it changes

Anthropic says disabling it prevents retrieving your content in response to a user query, "which may reduce your site's visibility for user-directed web search."

7%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: Claude-User
Disallow: /

How to allow it

User-agent: Claude-User
Allow: /

Good to know

Anthropic says its bots honor robots.txt.

Sources

What does your robots.txt say?

See how your site treats Claude-User and the other major crawlers.

Check my site

Other crawlers