Skip to Content
PersonaMark docs
Checks

robots.txt keeps AI search crawlers out

The site's robots.txt disallows a crawler that AI search products use to find pages to cite, like OAI-SearchBot for ChatGPT search or PerplexityBot, so those answers can't link to the site.

  • Area: Technical
  • Severity: Low
  • Check id: ai-search-blocked

How to fix it

If that isn't intended, remove the Disallow rule for that user agent. Blocking crawlers that only train models, like GPTBot or Google-Extended, doesn't affect this.

Where to see it

Open an audit's Issues tab and choose this check. Its page lists every page it found, lets you search them by address, mark issues as intended, and copy a prompt that files the fix in your tracker. Reading the issues explains what's on it.

On this page