AI crawlers ยท Perplexity

PerplexityBot

Used for: Perplexity search.

What it does

Perplexity says it's "designed to surface and link websites in search results on Perplexity," and "not used to crawl content for AI foundation models."

What blocking it changes

Perplexity recommends allowing it in robots.txt to appear in its search results.

11%

of the 228 sites we've audited with a readable robots.txt block it site-wide.

How to block it

User-agent: PerplexityBot
Disallow: /

How to allow it

User-agent: PerplexityBot
Allow: /

Good to know

Perplexity says changes may take up to 24 hours. Its separate Perplexity-User fetcher "generally ignores robots.txt rules," because a user requested the fetch.

Sources

What does your robots.txt say?

See how your site treats PerplexityBot and the other major crawlers.

Check my site

Other crawlers