Fix guides ยท Crawler access

Publish an XML sitemap

Publish an XML sitemap listing your URLs, and reference it from robots.txt.

Published standard

What we check

We fetch /sitemap.xml and look for a urlset or sitemapindex element. If it isn't there, we fetch the one your robots.txt declares. It warns if a declared sitemap can't be read, and fails if there's neither.

78%

of 198 sites we could read pass this.

Why it matters

Google says you can host a sitemap anywhere, but recommends the site root. A valid sitemap declared in robots.txt at another location passes this check.

Submitting one "is merely a hint: it doesn't guarantee that Google will download the sitemap."

Limits are 50MB or 50,000 URLs per sitemap, with absolute URLs in UTF-8. Google "ignores <priority> and <changefreq> values."

How to fix it

  1. Generate a sitemap. Most CMSes and frameworks can do this for you.
  2. Publish it, ideally at your site root.
  3. Add its location to robots.txt.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yoursite.com/</loc>
  </url>
</urlset>

Fix it yourself with AI

Small fixes shouldn't sit in a queue. Give your AI agent one prompt, and it fixes and deploys for you.

PromptShip is made by the team behind this checker.

Sources

Fixed it?

Run your site through the checker, and re-check after you ship to see what improved.

Check my site

Related guides