AI crawlers ยท Google

Googlebot

Used for: Google Search.

What it does

Google calls it "the generic name for two types of web crawlers used by Google Search," Smartphone and Desktop. Both use the same robots.txt token.

What blocking it changes

Google says blocking it "affects Google Search (including Discover and all Google Search features)," along with Images, Video and News.

0%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: Googlebot
Disallow: /

How to allow it

User-agent: Googlebot
Allow: /

Good to know

Blocking crawling doesn't remove a URL from results: "blocking Googlebot from crawling a page doesn't prevent the URL of the page from appearing in search results." Google's AI features rely on core Search.

Sources

What does your robots.txt say?

See how your site treats Googlebot and the other major crawlers.

Check my site

Other crawlers