Free tool

robots.txt Generator for AI Crawlers:
Allow AI Search, Block Training.

Pick a policy per role - AI search indexing, user-triggered fetches, model training - across 13 current agents (plus 2 legacy tokens handled for you), and get a copy-paste robots.txt block with an explanation of every line and a warning when a choice is self-defeating.

AI search indexing

Crawlers that index content so AI search surfaces (ChatGPT search, Claude search, Perplexity) can find and cite it. Blocking these removes you from those answers.

User-triggered fetches

Live fetches made when a real user question sends the assistant to your page. Blocking these breaks link-outs and live citations of your pages inside answers.

Model training + archival

Crawlers that collect content to train future models (plus control tokens like Google-Extended and Applebot-Extended, and Common Crawl archival that feeds many models). Blocking these does not remove you from AI search results.

Read before you paste - consequences of this selection

Google-Extended is a control token, not a crawler: per Google's docs it gates Gemini training/grounding use and "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".
CCBot feeds the Common Crawl corpus used by many model builders - blocking it is the broadest single training opt-out, and it also removes you from research datasets.

Your robots.txt block

# AI crawler rules - generated with citevera.com/tools/robots-txt-generator
# Paste into your robots.txt. Groups naming an agent override your
# wildcard (User-agent: *) group for that agent.

# AI search indexing: allowed (explicit, so a wildcard block cannot catch them)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: YouBot
Allow: /

# User-triggered fetches: allowed (explicit, so a wildcard block cannot catch them)
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /

# Model training + archival: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: cohere-ai
User-agent: Meta-ExternalAgent
Disallow: /

# Legacy tokens - not in current vendor docs but still honored;
# included so older conventions cannot bypass the block above.
User-agent: anthropic-ai
Disallow: /

What each line does (14 agents)

OAI-SearchBot (allow) - ChatGPT search index crawler. Explicit Allow protects it from wildcard Disallow rules elsewhere in your file.
ChatGPT-User (allow) - Live fetches during ChatGPT answers. Explicit Allow protects it from wildcard Disallow rules elsewhere in your file.
Claude-User (allow) - Live fetches during Claude answers. Explicit Allow protects it from wildcard Disallow rules elsewhere in your file.
Claude-SearchBot (allow) - Claude search index crawler. Explicit Allow protects it from wildcard Disallow rules elsewhere in your file.
PerplexityBot (allow) - Perplexity index crawler. Explicit Allow protects it from wildcard Disallow rules elsewhere in your file.
YouBot (allow) - You.com crawler. Explicit Allow protects it from wildcard Disallow rules elsewhere in your file.
GPTBot (block) - Model-training crawler. Training only - search surfacing is OAI-SearchBot, not GPTBot.
ClaudeBot (block) - Model-training crawler.
Google-Extended (block) - Gemini training opt-out token. A control token, not a fetcher. Per Google: gates use of crawled content for Gemini training and grounding; "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".
CCBot (block) - Common Crawl (feeds many models).
Applebot-Extended (block) - Apple Intelligence opt-out token.
cohere-ai (block) - Cohere crawler. Cohere publishes no dedicated crawler doc page; token observed in the wild.
Meta-ExternalAgent (block) - Meta AI crawler.
anthropic-ai (block) - Legacy Anthropic token, follows your Anthropic decision. No longer in Anthropic docs; still honored because older robots.txt files reference it.

Optional: compare against your current robots.txt

We fetch your live robots.txt and show, agent by agent, what your site says today next to what this selection would say.

How it works

Agents come from a maintained registry - the same list our crawler access checker tests and the WordPress plugin manages, each entry verified against vendor documentation (last verified 2026-08-04).
Grouped by role, not vendor - the decision that matters is search vs live fetches vs training, because those have opposite visibility consequences.
Warnings on self-defeating choices - blocking OAI-SearchBot while wanting ChatGPT visibility is the classic mistake; the tool says so before you paste, not after your citations disappear.
Optional live comparison - we fetch your current robots.txt and show today-vs-new per agent, with the current deciding rule quoted.

Example result

The "Allow AI search, block training" preset produces this block - real output of the same generator, rendered here statically. Note the explicit Allow groups (so a wildcard Disallow elsewhere in your file cannot catch the crawlers you want) and the labelled legacy-token section:

# AI crawler rules - generated with citevera.com/tools/robots-txt-generator (2026-08-04)
# Paste into your robots.txt. Groups naming an agent override your
# wildcard (User-agent: *) group for that agent.

# AI search indexing: allowed (explicit, so a wildcard block cannot catch them)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: YouBot
Allow: /

# User-triggered fetches: allowed (explicit, so a wildcard block cannot catch them)
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /

# Model training + archival: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: cohere-ai
User-agent: Meta-ExternalAgent
Disallow: /

# Legacy tokens - not in current vendor docs but still honored;
# included so older conventions cannot bypass the block above.
User-agent: anthropic-ai
Disallow: /

Common robots.txt mistakes with AI crawlers

Blocking OAI-SearchBot or ChatGPT-User to "stop AI" - and silently leaving ChatGPT answers unable to cite you. GPTBot is the training crawler.
Treating PerplexityBot as a training opt-out - it is an index crawler; blocking it removes you from Perplexity answers.
Relying on a wildcard group and assuming named agents inherit it - a group naming an agent overrides the wildcard entirely for that agent.
Blocking current tokens but not legacy ones (anthropic-ai, Claude-Web) - or the reverse, blocking only legacy tokens from an old template.
Expecting robots.txt to be enforcement - it is honored policy; a WAF is enforcement.
Forgetting the file is public - your robots.txt is readable by anyone, so rules are a statement of policy, not a secret.

Frequently asked questions

Can I block AI model training without disappearing from AI search?

Yes - that split is the whole point of per-agent rules. The training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) and the control tokens (Google-Extended, Applebot-Extended) are separate from the search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). The "Allow AI search, block training" preset expresses exactly this.

Why does the output use explicit Allow lines for agents I allow?

Because a robots.txt group naming an agent overrides your wildcard (User-agent: *) group for that agent. If your file has a broad Disallow anywhere, an agent with no named group can be caught by it. The explicit Allow group makes your intent unambiguous either way.

What are the legacy tokens in the output?

Tokens like anthropic-ai and Claude-Web are no longer in Anthropic's current docs but are still honored, because older robots.txt files reference them. When you block a vendor's current crawler, we include its legacy tokens with the same directive - labelled - so an old convention cannot bypass your block.

Does blocking Google-Extended hurt my Google rankings?

Per Google's crawler documentation, no: Google-Extended controls whether crawled content may be used for Gemini training and grounding, and Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It is a control token read by Googlebot, not a separate crawler.

Will these rules actually be obeyed?

robots.txt is a convention, not enforcement - but the vendors behind every agent in this generator document that they honor it, and each agent links to that documentation. For actual enforcement you need WAF/CDN rules; robots.txt is still the right first layer because it is the one every documented crawler reads.

After you paste the rules, verify them from the outside: run the AI crawler access checker and watch the per-agent verdicts match your new policy. On WordPress, the Citevera plugin manages the same per-agent rules from wp-admin. Background reading: the complete robots.txt guide for AI crawlers and should you allow AI crawlers.

Built by Paul, founder of Citevera · Published 2026-08-04 · Updated 2026-08-04 · Agent list verified 2026-08-04 against vendor docs · Matches the Citevera WordPress plugin

robots.txt Generator for AI Crawlers:Allow AI Search, Block Training.