bigbinhire.com.au

Can AI crawlers find your content and understand your policies?

• ✗
  robots.txt present

  Not found

  How to fix

  Add a robots.txt at your site root. Minimum: `User-agent: *\nAllow: /\nSitemap: https://yoursite.com/sitemap.xml`.

• ✗
  sitemap.xml present

  Not found

  How to fix

  Generate a sitemap.xml listing your canonical URLs (most CMSes do this automatically) and serve it at /sitemap.xml.

• !
  Sitemap declared in robots.txt

  Add a `Sitemap:` line to robots.txt

  How to fix

  Append a line like `Sitemap: https://yoursite.com/sitemap.xml` to robots.txt so crawlers find the sitemap immediately.

• –
  AI bot rules in robots.txt

  robots.txt missing
• !
  Content Signals declared

  No Cloudflare Content-Signal directive

  How to fix

  Add `Content-Signal: search=yes, ai-train=no, ai-input=yes` (tune the values to your policy) inside a User-agent group in robots.txt.

Standard well-known endpoints agents probe to interact with your site.

• ✗
  MCP server card (/.well-known/mcp/server-card.json)

  Publish an MCP server card to expose agent tools

  How to fix

  Publish an MCP server-card JSON at /.well-known/mcp/server-card.json describing your tools, transport, and capabilities. Spec: SEP-1649/SEP-2127.

• ✗
  API catalog (/.well-known/api-catalog)

  RFC 9727 API catalog helps agents discover your APIs

  How to fix

  Serve a `application/linkset+json` document at /.well-known/api-catalog listing your OpenAPI / service descriptions. RFC 9727.

• ✗
  OAuth protected resource (/.well-known/oauth-protected-resource)

  RFC 9728 lets agents discover auth requirements

  How to fix

  Publish a JSON document at /.well-known/oauth-protected-resource with `resource` and `authorization_servers` fields. RFC 9728.

• ✗
  OpenID configuration (/.well-known/openid-configuration)

  Helpful if your site offers identity / sign-in for agents

  How to fix

  If your product supports sign-in, expose OpenID Connect discovery at /.well-known/openid-configuration so agents can authenticate users on their behalf.

How well your site speaks formats AI agents prefer.

• ✗
  llms.txt present

  Add /llms.txt to give models a curated entry point

  How to fix

  Create /llms.txt as Markdown with a short site description, key URLs, and your content/training policy. See llmstxt.org for the spec.

• ✗
  Markdown content negotiation

  Server didn't honor Accept: text/markdown

  How to fix

  Implement HTTP content negotiation: when the request's `Accept` header is `text/markdown`, return a Markdown representation of the page.

• !
  Link header on /

  No Link header — agents can't follow alt-format relations

  How to fix

  Add an HTTP `Link:` header pointing agents at alternate representations, e.g. `Link: </openapi.json>; rel="service-desc"; type="application/openapi+json"`.

HTML signals that Google AI Search + LLM crawlers use to rank you.

• ✗
  Homepage reachable

  HTTP 403

  How to fix

  The audit couldn't fetch your homepage. Check that the site is up, that the SSL certificate is valid, and that you don't block our User-Agent.

Optional: cryptographic verification for trusted bots.

• ✗
  Web Bot Auth directory

  Publish /.well-known/http-message-signatures-directory to verify trusted bots

  How to fix

  If you operate a bot, publish your signing keys at /.well-known/http-message-signatures-directory per draft-meunier-web-bot-auth-architecture. Most non-bot sites can leave this missing.