# robots.txt for AI crawlers: copy-paste examples with explanations

> Three tested robots.txt policies for AI crawlers — allow all, block training but stay in AI answers, and block everything — plus the mistakes that break them.

Canonical: https://crawlwise.site/guides/ai-search/robots-txt-for-ai-crawlers-examples
Last updated: 2026-11-05

Pick one of three policies. To allow everything, a plain User-agent: \* group is enough. To block training but stay in AI answers, disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and similar tokens while allowing the search agents. To block everything, name every AI agent and enforce at your CDN for fetchers that ignore robots.txt.

by Adnan Arodiya

Updated November 5, 2026 · 6 min read

## Key takeaways

- Several User-agent lines can share one group of rules.
- A named group replaces the \* group for that crawler; repeat shared disallows.
- Tokens like Google-Extended only work in robots.txt, never at the firewall.
- Test the live file after every change.

These files use only user-agent tokens that the vendors document. Replace the sitemap URL and any private paths with your own, then test the live result. Why each crawler is in each list is covered in [which AI crawlers should you allow?](https://crawlwise.site/guides/ai-search/which-ai-crawlers-to-allow)

## How do you allow every crawler?

```
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
```

No AI crawler is named, so every one of them follows the `*` group. This is also what a missing robots.txt (a 404) means, minus the `/admin/` rule.

## How do you block AI training but stay in AI answers?

```
# Training crawlers and training-control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /

# AI search crawlers and user-triggered fetchers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
```

Notes on the choices. **Google-Extended** and **Applebot-Extended** keep Googlebot and Applebot crawling for Search, Siri and Spotlight while opting out of Gemini and Apple model training; Google-Extended also opts out of Gemini Apps grounding. **Meta-ExternalAgent** is mixed purpose (training or indexing, per Meta), so blocking it may cost some Meta product visibility. **CCBot** is an open archive many others use; blocking it is the widest single training opt-out. The `/admin/` line is repeated in the search group because a named group does not inherit the `*` rules.

> A named group replaces the star group. If a rule matters, write it in every group that should obey it.
>
> Adnan Arodiya, Crawlwise

## How do you block every AI crawler?

```
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Disallow: /admin/
```

This removes you from ChatGPT search, Claude’s and Perplexity’s indexes, and opts out of training, while keeping Googlebot and Bingbot for classic search. It does not keep you out of Google AI Overviews or Copilot answers, which are built on the regular Google and Bing indexes; those need snippet controls (`nosnippet` for Google, `NOARCHIVE` or `NOCACHE` for Bing). User-triggered fetchers may ignore the file, so add a CDN rule if you need a hard block.

## Which mistakes break these files?

- **Forgetting the group rule.** Adding `User-agent: GPTBot` with only `Allow: /` opens every path you disallowed under `*` to GPTBot.
- **Blocking at the firewall by token.** Google-Extended and Applebot-Extended never appear in requests; a WAF rule for them matches nothing.
- **Serving robots.txt behind a challenge.** If the file returns an HTML challenge, crawlers cannot read your policy at all.
- **Using robots.txt to remove pages.** It controls fetching, not indexing.

## What do popular sites’ files look like?

In the [Crawlwise AI Crawler Access Report](https://crawlwise.site/reports/ai-crawler-access) (top-10000 run, 2026-Q4: the first 10,000 domains of the Cisco Umbrella top-one-million list, each probed once on November 4, 2026), of the 1,874 domains with a readable robots.txt, 711 (37.9%) disallow all 17 AI crawlers we test at the homepage — almost always a blanket `Disallow: /` — while only 145 (7.7%) give different crawlers different answers. Policies like the middle example are still rare. The list ranks hostnames by DNS popularity, so it includes API, CDN and update hosts that never serve a web page, and only the homepage path was probed. Treat the shares as a description of that list, not of the whole web.

## How Crawlwise tests this

Generate a file with the [robots.txt generator](https://crawlwise.site/tools/robots-txt), then paste a path and crawler into the [robots.txt tester](https://crawlwise.site/tools/robots-tester) to see which group and rule decide it. After you deploy, run the [AI crawler checker](https://crawlwise.site/tools/ai-crawler-check) to confirm the live result, including any CDN override.

Every verdict, what counts as a block, and what the checks cannot see are written up on the [methodology page](https://crawlwise.site/methodology).

## What a plan actually costs

The free tools stay free. You can open them, run them, and leave without an account. A plan is for the moment you want the report saved, a few sites watched, or the same checks from the API. The price on this table is the price at checkout. A yearly plan is ten months of the monthly price, so two months are on us.

| Plan | Monthly | Credits | A good fit when |
| --- | --- | --- | --- |
| Starter | $14.99 | 40 | Regular audits of up to three of your own sites, with Search Console and watched URLs |
| Growth | $49.99 | 150 | Client work: backlinks, keyword research, rank tracking and the API |
| Agency | $99.99 | 400 | Many locations, reported under your own name |

A single-page audit is about 1.25 credits, and that includes the live probe of answer-engine crawlers. You see the estimate before anything runs. If a hold is not used, it comes back to your balance. Extra credits, when you already subscribe, are $9.99 for 20.

## Frequently asked questions

### Can I list several user agents in one group?

Yes. RFC 9309 allows several User-agent lines before one set of rules. Every listed crawler follows that group.

### Does Crawl-delay work for AI crawlers?

It is not part of RFC 9309. Anthropic says it supports Crawl-delay; Google ignores it. Check each vendor’s docs before relying on it.

### Will these rules stop user-triggered fetchers?

Not reliably. OpenAI, Perplexity, Google and Meta all say their user-triggered fetchers may ignore robots.txt. Block those at your server or CDN if you need to.

## Sources

Vendor documentation was read on 2026-10-05. Vendors change these pages without notice, so check the original before you act on a detail.

- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [Google Search Central — How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- [OpenAI — Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity — Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Google Search Central — Google common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- [Apple — About Applebot](https://support.apple.com/en-us/119829)
- [Meta — Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
- [Common Crawl — CCBot](https://commoncrawl.org/ccbot)
- [Bing Webmaster Blog — New options for webmasters to control usage of their content in Bing Chat (Sept 2023)](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat)
- [Crawlwise — The AI Crawler Access Report (top-10000 run, 2026-Q4)](https://crawlwise.site/reports/ai-crawler-access)

## Crawlwise crawler check vs checking by hand

| What you get | Crawlwise | By hand |
| --- | --- | --- |
| Resolves robots.txt groups the way crawlers do (RFC 9309) | Yes, per crawler token | By eye; group precedence is easy to misread |
| Fetches the page as each AI crawler | Yes, one live request per agent | One curl at a time |
| Spots CDN and firewall blocks (403, challenge pages) | Yes, with challenge fingerprints | Only if you read the response body |
| Separates a training opt-out from an accidental block | Labelled separately | Up to you |
| Sees blocks that only happen in other regions or on real crawler IPs | No — one location, one moment | Only from your own server logs |

## Meet the author

Adnan Arodiya

Crawlwise

Writes about what Crawlwise actually measures: on-page evidence, crawler access, and performance signals — with the limits stated up front.

## Related guides

[__usage_ai-training: the one robots.txt line to opt out of AI training](https://crawlwise.site/guides/ai-search/shared-ai-training-opt-out-robots-txt): A single robots.txt group now states your AI-training preference to every provider that honours it. Here is the syntax and the catch with Google-Extended.

[What is GEO? Generative engine optimization vs SEO vs AEO, explained simply](https://crawlwise.site/guides/ai-search/what-is-geo): GEO, AEO and SEO describe overlapping jobs. Here is what is genuinely new, and what is the same work with a new name.

[What is query fan-out, and why no tool can show you Google’s real fan-out queries](https://crawlwise.site/guides/ai-search/what-is-query-fan-out): AI search splits one question into many searches. Google does not publish them, so here is what you can and cannot know.
