How we did this
- The list: we started from the Tranco ranking of 2026-11-08 and took 100 sites, down to rank 336. Tranco is full of infrastructure like CDNs and DNS hosts, so we picked public websites by hand and left out adult sites. That selection is ours, so this isn't a random sample.
- The data: we audited 100 of them and could read a robots.txt policy for 92. The other 8 didn't give us a readable answer. A site with no robots.txt counts as allowing everything.
- AI search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. These are the headline numbers.
- On-request fetchers: ChatGPT-User and Claude-User fetch pages when a person asks. They're charted but not counted as search crawlers, because OpenAI says ChatGPT-User isn't used to determine what appears in Search.
- Training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot.
- The limits: this covers stated robots.txt policy only. A firewall can block a crawler robots.txt allows, and not every crawler honors robots.txt. Blocking can also be a deliberate choice, so it isn't a mistake by default. We report totals only and don't name individual sites here. Figures reflect each site's latest audit and can change.