robots.txt keeps AI search crawlers out
The site's robots.txt disallows a crawler that AI search products use to find pages to cite, like OAI-SearchBot for ChatGPT search or PerplexityBot, so those answers can't link to the site.
- Area: Technical
- Severity: Low
- Check id:
ai-search-blocked
How to fix it
If that isn't intended, remove the Disallow rule for that user agent. Blocking crawlers that only train models, like GPTBot or Google-Extended, doesn't affect this.
Where to see it
Open an audit's Issues tab and choose this check. Its page lists every page it found, lets you search them by address, mark issues as intended, and copy a prompt that files the fix in your tracker. Reading the issues explains what's on it.
Images that are heavier than they need to be
These pages load images over 200 KB, which slow the page down, especially on phones, and count against Core Web Vitals.
Pages aren't set up for phones
Google's Lighthouse found no viewport meta tag, so phones show these pages zoomed out, and Google treats them as not mobile-friendly. Our own viewport check covers every crawled page, not only these.