What it does
Common Crawl is a non-profit producing "an open repository of web crawl data." We list it under training because AI developers are among those who can use open crawl data, though its page doesn't say that.
What blocking it changes
Common Crawl gives robots.txt as the opt-out. Its page doesn't describe further effects.
16%
of the 228 sites we've audited with a readable robots.txt block it site-wide.
How to block it
User-agent: CCBot
Disallow: /
How to allow it
User-agent: CCBot
Allow: /
Good to know
It warns of "crawlers falsely identifying themselves as CCBot." Real CCBot IPs resolve to *.crawl.commoncrawl.org.
Sources
What does your robots.txt say?
See how your site treats CCBot and the other major crawlers.
Check my site