AI Crawler Access Checker:
Can ChatGPT See Your Page?
One URL in, one row per AI crawler out - 13 current agents plus 2 legacy tokens, each with the exact robots.txt rule that decides it, page-level noindex/noai signals, and one real fetch that detects WAF and Cloudflare challenges.
The robots.txt and page-signal rows report what your site declares. Our real fetch runs as Citevera-Scanner, not as each vendor's bot - so "allowed" means not blocked by policy, not "verified fetch as GPTBot". Per-vendor verified fetching is a different, harder measurement; no free checker does it, including this one - we just say so.
What this tests
- Declared robots policy - your robots.txt, parsed per agent with standard precedence, shown in the verdict column with the exact Allow/Disallow line that decided it.
- Page-level signals - meta robots tags and X-Robots-Tag headers (noindex/noai variants), which apply on top of robots.txt and appear in the deciding-rule column when present.
- One real fetch - the page requested with a crawler user-agent, reported in its own card: status, redirect behavior, and WAF/Cloudflare challenge detection.
Crawling, AI search, and model training are different
The big vendors run separate agents for separate jobs, and per-agent rules only make sense once you know which is which:
- GPTBot - Model-training crawler
- OAI-SearchBot - ChatGPT search index crawler
- ChatGPT-User - Live fetches during ChatGPT answers
- ClaudeBot - Model-training crawler
- Claude-User - Live fetches during Claude answers
- Claude-SearchBot - Claude search index crawler
Blocking a training crawler does not block search visibility, and vice versa. For every agent's role and vendor documentation, see the full AI crawler agent guide.
Example result
A real check of a major news publisher's homepage, run with this tool on 2026-08-04 (domain removed). 5 of 13 current crawlers allowed by policy - the publisher blocks Anthropic, Perplexity, Apple, Meta, and Common Crawl by name while leaving OpenAI's agents unmatched by any rule:
| Crawler | Verdict |
|---|---|
| GPTBot | allowed (no rule) |
| OAI-SearchBot | allowed (no rule) |
| ChatGPT-User | allowed (no rule) |
| ClaudeBot | blocked by robots.txt |
| Claude-User | blocked by robots.txt |
| Claude-SearchBot | blocked by robots.txt |
| PerplexityBot | blocked by robots.txt |
| Google-Extended | allowed (no rule) |
| CCBot | blocked by robots.txt |
| Applebot-Extended | blocked by robots.txt |
| YouBot | blocked by robots.txt |
| cohere-ai | allowed (no rule) |
| Meta-ExternalAgent | blocked by robots.txt |
Common reasons crawlers can't access a page
- A wildcard User-agent: * / Disallow: / blocking everything.
- A vendor-specific Disallow naming the agent (the deciding rule we quote).
- Page-level noindex or noai in meta robots or the X-Robots-Tag header.
- Cloudflare or WAF bot protection challenging crawler user-agents.
- An authentication or cookie wall in front of the content.
- A redirect loop the crawler abandons.
- 5xx errors or timeouts under crawler load.
- CDN configurations serving different content to bots than to browsers.
How to fix access
On WordPress: the Citevera plugin manages per-agent Allow/Disallow rules from wp-admin - the same agent list this checker tests.
Editing robots.txt directly: fix the specific rule this checker quotes. The most specific matching path wins, Allow beats Disallow at equal length, and a group naming an agent overrides the wildcard group.
Cloudflare / WAF: review AI Crawl Control or your bot rules and explicitly allow the crawlers you want. Our scanner's allowlist details are on the bot policy page.
SEO plugin settings: page-level noindex is often set per-post by an SEO plugin - check the page's advanced settings before editing headers by hand.
Frequently asked questions
If everything says "allowed", does that prove ChatGPT indexed my page?
No. "Allowed" means your declared policy (robots.txt and page signals) does not block the crawler - it is permission, not proof of a visit or of indexing. We fetch as Citevera-Scanner, not as each vendor's bot, so we report what your site declares plus how it answered one real crawler-like request.
Which crawler does ChatGPT Search use?
OAI-SearchBot indexes content for ChatGPT's search features. GPTBot is the model-training crawler, and ChatGPT-User makes live fetches when a user's question sends ChatGPT to a page. Three agents, three different jobs.
Can I allow AI search but block model training?
Yes - that is the point of the role split. Allow the search and user-triggered agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User) and disallow the training crawlers (GPTBot, ClaudeBot) with per-agent robots.txt rules.
What does Google-Extended actually control?
Per Google's crawler documentation, Google-Extended manages whether crawled content "may be used for training future generations of Gemini models" and for grounding. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It is a control token, not a crawler - Googlebot does the fetching.
Why does the page work in my browser but block crawlers?
Bot protection decides by user-agent and bot score, not by whether the page exists. Your browser presents a human fingerprint; a crawler user-agent presenting none gets the challenge page. That is why our checker makes a real crawler-like fetch instead of trusting what a browser sees.
Crawler access is step one; step two is giving those crawlers a curated index - check that your llms.txt actually serves the way a crawler requests it.
Built by Paul, founder of Citevera · Published 2026-08-04 · Updated 2026-08-04 · Agent list verified 2026-08-04 against vendor docs · Matches the Citevera WordPress plugin · Background: AI crawler user-agents in 2026