AI crawlers ยท Common Crawl

CCBot

Used for: Open web archive.

What it does

Common Crawl is a non-profit producing "an open repository of web crawl data." We list it under training because AI developers are among those who can use open crawl data, though its page doesn't say that.

What blocking it changes

Common Crawl gives robots.txt as the opt-out. Its page doesn't describe further effects.

16%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: CCBot
Disallow: /

How to allow it

User-agent: CCBot
Allow: /

Good to know

It warns of "crawlers falsely identifying themselves as CCBot." Real CCBot IPs resolve to *.crawl.commoncrawl.org.

Sources

What does your robots.txt say?

See how your site treats CCBot and the other major crawlers.

Check my site

Other crawlers