AI crawlers ยท Google

Google-Extended

Used for: Gemini training and grounding.

What it does

A robots.txt control token rather than a separate crawler: Google says it "doesn't have a separate HTTP request user agent string." It controls whether content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI.

What blocking it changes

Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

12%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: Google-Extended
Disallow: /

How to allow it

User-agent: Google-Extended
Allow: /

Good to know

You won't see it in your server logs, because it isn't a separate crawler.

Sources

What does your robots.txt say?

See how your site treats Google-Extended and the other major crawlers.

Check my site

Other crawlers