Fix guides ยท Crawler access

Fix a homepage that blocks automated requests

Make sure your homepage returns a 200 to automated requests, and check that your firewall or CDN isn't challenging our user agent.

Our audit requirement

What we check

We request your homepage over HTTPS. Anything other than a normal page counts as a failure, including 401, 403, 429 and 503 responses, bot-protection challenges, and no response. We then report "Couldn't audit" rather than a score.

100%

of 198 sites we could read pass this.

Why it matters

We can only measure what we can read, so this check is about whether an audit is possible at all. A block on our bot doesn't prove that AI crawlers are blocked too, since firewalls can treat each one differently.

How to fix it

  1. Load your homepage from a network outside your own and confirm it responds.
  2. Check your firewall, CDN or bot-protection rules for anything challenging automated traffic.
  3. If you want us to audit the site, allow the user agent AI-Visibility-Checker/1.0.
  4. Check that your certificate is valid for both your bare domain and www.

Fix it yourself with AI

Small fixes shouldn't sit in a queue. Give your AI agent one prompt, and it fixes and deploys for you.

PromptShip is made by the team behind this checker.

Fixed it?

Run your site through the checker, and re-check after you ship to see what improved.

Check my site

Related guides