Robots.txt AI Analyzer
Check which AI crawlers your robots.txt blocks or allows. See if ChatGPT, Claude, Perplexity, and other AI engines can access your content.
AI-Friendly robots.txt Template
Copy this template and replace the sitemap URL with your own.
User-agent: *
Allow: /
# AI Crawlers Welcome
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xml
How robots.txt affects AI visibility
AI crawlers read your robots.txt first
When GPTBot, ClaudeBot, or other AI crawlers visit your site, the first file they check is /robots.txt. If they find a Disallow rule targeting them, they will not crawl your content.
Blocking AI crawlers = invisible to AI search
If you block GPTBot, your content cannot appear in ChatGPT responses. If you block PerplexityBot, Perplexity cannot cite your site. Each blocked crawler is an AI audience you lose.
No robots.txt = everything is allowed
If your site has no robots.txt file, all crawlers (including AI crawlers) are allowed by default. Having an explicit robots.txt that welcomes AI crawlers gives you more control and signals intent.
Robots.txt is just one of 34 AEO factors
AI crawler access is one of the technical foundations that determines whether AI engines can find and recommend your content. But it is just one factor. Schema markup, content structure, authority signals, and more all play a role in your overall AI visibility score.
Want to check all 34 AI visibility factors?
Robots.txt is just one piece of the puzzle. Run a free AEO audit to see your full AI visibility score across all 34 factors.
Run a Free AEO Audit