AI crawlers ยท OpenAI

GPTBot

Used for: Model training.

What it does

OpenAI's training crawler.

What blocking it changes

OpenAI says disallowing it "indicates a site's content should not be used in training generative AI foundation models."

13%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: GPTBot
Disallow: /

How to allow it

User-agent: GPTBot
Allow: /

Good to know

Search inclusion is controlled separately, through OAI-SearchBot.

Sources

What does your robots.txt say?

See how your site treats GPTBot and the other major crawlers.

Check my site

Other crawlers