AI crawlers ยท Apple

Applebot-Extended

Used for: Apple AI training.

What it does

A control token rather than a crawler: Apple says it "does not crawl webpages," and is "only used to determine how to use the data crawled by the Applebot user agent."

What blocking it changes

Apple says pages that disallow it "can still be included in search results." It opts content out of training Apple's general purpose foundation models.

14%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: Applebot-Extended
Disallow: /

How to allow it

User-agent: Applebot-Extended
Allow: /

Good to know

Apple says rules for it "are not considered in ranking for Search."

Sources

What does your robots.txt say?

See how your site treats Applebot-Extended and the other major crawlers.

Check my site

Other crawlers