AI crawlers ยท OpenAI

OAI-SearchBot

Used for: ChatGPT search.

What it does

OpenAI says it's "used to surface websites in search results in ChatGPT's search features."

What blocking it changes

OpenAI says sites opted out "will not be shown in ChatGPT search answers," though they "can still appear as navigational links."

7%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: OAI-SearchBot
Disallow: /

How to allow it

User-agent: OAI-SearchBot
Allow: /

Good to know

OpenAI says robots.txt changes can take about 24 hours to take effect for search.

Sources

What does your robots.txt say?

See how your site treats OAI-SearchBot and the other major crawlers.

Check my site

Other crawlers