State of AI Readiness · Live data from real audits

Don't get outcompeted in AI search

What 301 websites look like to AI crawlers, and where they most often fall short.

Sites people check here are already interested in AI search, so treat this as directional rather than a picture of the whole web. How we measure this

How many top sites block AI search?

We read the robots.txt of 100 top-ranked websites to see which AI crawlers they turn away.

Block at least one AI search crawler
23%
21 of 92 sites
Limit AI search to a few paths
7%
6 of 92 sites
Block at least one training crawler
34%
31 of 92 sites

Share of sites blocking each crawler

CCBottraining 29%
ClaudeBottraining 25%
Applebot-Extendedtraining 25%
PerplexityBotsearch 23%
GPTBottraining 23%
Google-Extendedtraining 22%
Claude-SearchBotsearch 17%
ChatGPT-Useron request 17%
OAI-SearchBotsearch 15%
Claude-Useron request 15%

Share of 92 sites blocking the crawler across the whole site. A limited crawler isn't counted as blocked.

How we did this

  • The list: we started from the Tranco ranking of 2026-11-08 and took 100 sites, down to rank 336. Tranco is full of infrastructure like CDNs and DNS hosts, so we picked public websites by hand and left out adult sites. That selection is ours, so this isn't a random sample.
  • The data: we audited 100 of them and could read a robots.txt policy for 92. The other 8 didn't give us a readable answer. A site with no robots.txt counts as allowing everything.
  • AI search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. These are the headline numbers.
  • On-request fetchers: ChatGPT-User and Claude-User fetch pages when a person asks. They're charted but not counted as search crawlers, because OpenAI says ChatGPT-User isn't used to determine what appears in Search.
  • Training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot.
  • The limits: this covers stated robots.txt policy only. A firewall can block a crawler robots.txt allows, and not every crawler honors robots.txt. Blocking can also be a deliberate choice, so it isn't a mistake by default. We report totals only and don't name individual sites here. Figures reflect each site's latest audit and can change.

Traditional sectors

147 sites. We could read 72; the other 75 blocked our bot or didn't respond, and aren't included below.

Big companies in sectors that aren't primarily tech, chosen by rank before we scanned them. How we chose them

Could be read
72 of 147
The rest blocked our bot or didn't respond
Scored under 50
14%
10 of 72 sites read
Median score
90
Only sites that let us in, out of 100

Median score by sector

Real estate7 sites read 97
Insurance12 sites read 95
Automotive6 sites read 95
Hotels7 sites read 94
Banking15 sites read 93
Healthcare15 sites read 90
Travel and airlines10 sites read 81

Each sector has few sites read, so small differences don't mean much.

By sector

SectorSites readMedian scoreBlock AI searchLittle text without JSNo structured dataHave llms.txt
Real estate7970%14%43%14%
Travel and airlines10810%30%70%0%
Hotels7940%0%43%29%
Insurance12950%17%42%42%
Banking15930%13%47%13%
Healthcare15900%20%40%33%
Automotive69517%0%33%33%
Median headline score
90
Crawler access and fundamentals
Median crawler access
100
Out of 100
Median fundamentals
88
Out of 100

Headline score distribution

0 0–9 0 10–19 0 20–29 1 30–39 9 40–49 1 50–59 1 60–69 3 70–79 16 80–89 41 90–100
View as table
Score bandSites
0–90
10–190
20–290
30–391
40–499
50–591
60–691
70–793
80–8916
90–10041

Site profiles

Blocks AI searchrobots.txt tells at least one AI search crawler to stay out. 1
Thin without JavaScriptThe homepage has under 30 words until JavaScript runs. 8
Early adopterAdopted 3 or more emerging standards. 1
Solid basicsFundamentals of 80 or more, with few emerging standards. 47
Basics gapCrawlers get in, but fundamentals score under 80. 15

Share of sites passing each check

Crawler access

Homepage reachable 100%
HTTPS 100%
AI crawler access 99%
Readable without JavaScript 85%
robots.txt present 83%
sitemap.xml present 76%
Sitemap declared in robots.txt 67%

Search fundamentals

Title tag 94%
Mobile viewport 94%
html lang attribute 92%
Image alt text 89%
Meta description 82%
Canonical URL 82%
Open Graph tags 57%
Schema.org structured data 54%

Emerging standards

llms.txt present 24%
Link header on / 4%
Content Signals declared 3%
Markdown content negotiation 3%
API catalog 1%
MCP server card 1%
OAuth protected resource 0%
OpenID configuration 0%
Web Bot Auth directory 0%

Which crawlers robots.txt blocks

CCBotCommon Crawl · training 7%
Applebot-ExtendedApple · training 4%
ClaudeBotAnthropic · training 4%
GPTBotOpenAI · training 4%
Google-ExtendedGoogle · training 4%
PerplexityBotPerplexity · search 1%
BingbotMicrosoft · search 0%
ChatGPT-UserOpenAI · search 0%
Claude-SearchBotAnthropic · search 0%
Claude-UserAnthropic · search 0%
GooglebotGoogle · search 0%
OAI-SearchBotOpenAI · search 0%

Share of 71 sites with a readable policy.

Checks that tend to pass together

If a site passesThen it also passesWithWithout
Canonical URLhtml lang attribute100%54%
robots.txt presentsitemap.xml present88%17%
Readable without JavaScriptMeta description92%27%
Meta descriptionCanonical URL90%46%
Readable without JavaScripthtml lang attribute97%64%
sitemap.xml presentMeta description91%53%

Correlation only, across 72 sites. Neither check causes the other.

Sites people checked

60 sites. We could read 54; the other 6 blocked our bot or didn't respond, and aren't included below.

1 of these audits predates our stricter checks and will update as sites are re-audited, so some passes below may not hold.

Median headline score
100
Crawler access and fundamentals
Median crawler access
100
Out of 100
Median fundamentals
100
Out of 100

Headline score distribution

0 0–9 0 10–19 0 20–29 0 30–39 4 40–49 0 50–59 0 60–69 2 70–79 2 80–89 46 90–100
View as table
Score bandSites
0–90
10–190
20–290
30–390
40–494
50–590
60–690
70–792
80–892
90–10046

Site profiles

Blocks AI searchrobots.txt tells at least one AI search crawler to stay out. 2
Thin without JavaScriptThe homepage has under 30 words until JavaScript runs. 2
Early adopterAdopted 3 or more emerging standards. 13
Solid basicsFundamentals of 80 or more, with few emerging standards. 33
Basics gapCrawlers get in, but fundamentals score under 80. 4

Share of sites passing each check

Crawler access

Homepage reachable 100%
HTTPS 100%
robots.txt present 98%
AI crawler access 94%
sitemap.xml present 93%
Sitemap declared in robots.txt 93%
Readable without JavaScript 91%

Search fundamentals

Title tag 98%
Mobile viewport 98%
html lang attribute 98%
Canonical URL 96%
Meta description 91%
Image alt text 88%
Open Graph tags 81%
Schema.org structured data 80%

Emerging standards

llms.txt present 67%
Link header on / 28%
Content Signals declared 17%
OAuth protected resource 17%
Markdown content negotiation 15%
MCP server card 13%
OpenID configuration 13%
API catalog 7%
Web Bot Auth directory 0%

Which crawlers robots.txt blocks

ClaudeBotAnthropic · training 10%
GPTBotOpenAI · training 10%
Google-ExtendedGoogle · training 10%
Applebot-ExtendedApple · training 8%
CCBotCommon Crawl · training 8%
ChatGPT-UserOpenAI · search 4%
Claude-SearchBotAnthropic · search 4%
Claude-UserAnthropic · search 4%
OAI-SearchBotOpenAI · search 4%
PerplexityBotPerplexity · search 4%
BingbotMicrosoft · search 0%
GooglebotGoogle · search 0%

Share of 52 sites with a readable policy.

Checks that tend to pass together

If a site passesThen it also passesWithWithout
Content Signals declaredMCP server card67%2%
Meta descriptionOpen Graph tags90%0%
Content Signals declaredMarkdown content negotiation67%4%
Markdown content negotiationMCP server card62%4%
Markdown content negotiationLink header on /88%17%
Readable without JavaScriptSchema.org structured data88%20%

Correlation only, across 54 sites. Neither check causes the other.

Top-ranked sites (Tranco)

94 sites. We could read 72; the other 22 blocked our bot or didn't respond, and aren't included below.

Median headline score
86
Crawler access and fundamentals
Median crawler access
90
Out of 100
Median fundamentals
88
Out of 100

Headline score distribution

0 0–9 0 10–19 0 20–29 3 30–39 23 40–49 0 50–59 1 60–69 2 70–79 14 80–89 29 90–100
View as table
Score bandSites
0–90
10–190
20–290
30–393
40–4923
50–590
60–691
70–792
80–8914
90–10029

Site profiles

Blocks AI searchrobots.txt tells at least one AI search crawler to stay out. 15
Thin without JavaScriptThe homepage has under 30 words until JavaScript runs. 11
Early adopterAdopted 3 or more emerging standards. 2
Solid basicsFundamentals of 80 or more, with few emerging standards. 36
Basics gapCrawlers get in, but fundamentals score under 80. 8

Share of sites passing each check

Crawler access

Homepage reachable 100%
HTTPS 100%
robots.txt present 93%
AI crawler access 74%
Sitemap declared in robots.txt 72%
Readable without JavaScript 69%
sitemap.xml present 68%

Search fundamentals

Title tag 93%
Mobile viewport 88%
html lang attribute 88%
Image alt text 85%
Meta description 81%
Canonical URL 75%
Open Graph tags 56%
Schema.org structured data 40%

Emerging standards

llms.txt present 19%
OpenID configuration 11%
Link header on / 4%
MCP server card 4%
Content Signals declared 1%
Markdown content negotiation 1%
API catalog 1%
Web Bot Auth directory 1%
OAuth protected resource 0%

Which crawlers robots.txt blocks

CCBotCommon Crawl · training 29%
ClaudeBotAnthropic · training 24%
Applebot-ExtendedApple · training 22%
GPTBotOpenAI · training 21%
Google-ExtendedGoogle · training 21%
PerplexityBotPerplexity · search 21%
Claude-SearchBotAnthropic · search 17%
ChatGPT-UserOpenAI · search 15%
Claude-UserAnthropic · search 14%
OAI-SearchBotOpenAI · search 14%
BingbotMicrosoft · search 1%
GooglebotGoogle · search 0%

Share of 72 sites with a readable policy.

Checks that tend to pass together

If a site passesThen it also passesWithWithout
Mobile viewporthtml lang attribute97%22%
Meta descriptionCanonical URL88%21%
Readable without JavaScriptMeta description96%45%
Title tagMeta description87%0%
Title tagMobile viewport93%20%
Title taghtml lang attribute93%20%

Correlation only, across 72 sites. Neither check causes the other.

By category

CategorySitesMedian
Software & SaaS11100
Retail & consumer brands1197
Automotive & mobility796
Professional & local services694
Travel, hospitality & events5100
Finance & fintech4100
Marketing agencies4100

Now check your competition.

Are your competitors open to AI search? Are you? Start with any site.