IntentNexus

Start Free Trial

AI Crawlability Check

See which AI bots can actually read your site in 15 seconds.

Drop in your domain. We'll fetch your robots.txt, llms.txt, sitemap, and canonical setup — and tell you which of GPTBot, ClaudeBot, PerplexityBot, and Googlebot are allowed access. Plus the exact robots.txt fixes when something's blocked.

No signup · 15 seconds · Up to 10 audits per hour

RUN AUDIT

What your score actually means.

90–100

Fully crawlable

Every major AI bot is allowed and your structural files (robots.txt, llms.txt, sitemap) are clean. AI engines can index your content the same way Google does. Maintain by checking every quarter or whenever you change CMS settings.

50–89

Partially blocked

Googlebot can read your site but at least one of GPTBot, ClaudeBot, or PerplexityBot is denied or unaddressed. This is the most common state — most CMSs default-block AI bots. A single robots.txt edit unblocks them.

0–49

Largely blocked

AI bots are blocked at the robots.txt level or your sitemap is invalid. Until this is fixed, AI engines literally cannot read your content — which means they cannot cite it. The fixes section below shows the exact lines to add.

Crawlability fixes are step one. Tracking citations is step two.

This audit confirms AI bots can read you. The IntentNexus platform measures whether they actually cite you when your customers ask — across ChatGPT, Perplexity, Google AI, and Copilot — and shows you the questions where competitors are cited instead.

See How the Platform Works →
Or run another free audit →

Run another audit to dig deeper.

GEO Visibility Audit

predict your overall citation potential

Run audit →

Schema Readiness Audit

find missing structured-data signals

Run audit →

Content Authority Score

measure your E-E-A-T trust signals

Run audit →

About this audit.

Why might GPTBot be blocked when Googlebot is allowed?
Most CMS robots.txt defaults block GPTBot, ClaudeBot, and PerplexityBot by name while leaving Googlebot allowed. WordPress, Shopify, and Webflow have all done this in different versions. The fix is one line in robots.txt: User-agent: GPTBot followed by Allow: /.
Do I need an llms.txt file?
It is not required, but it is increasingly recommended. llms.txt is a community standard for telling AI engines which content you want surfaced. Anthropic, OpenAI, and Perplexity have all signaled support. Adding one is a small upgrade that signals AI-readiness.
How often should I re-check?
Quarterly is enough for most sites. Re-check whenever you migrate hosts, change CMS versions, or adjust robots.txt. New AI bots also appear periodically — running this audit catches new bots you have not addressed yet.
Will allowing AI bots cost me Google ranking?
No. Allowing GPTBot is independent of Googlebot ranking. Google’s guidance is explicit that AI bot access does not affect search ranking, and AI citations are a separate channel that compounds with Google traffic.