# AI crawler statistics: the AI Crawler Access Report

> AI crawler statistics: how many websites block GPTBot, ClaudeBot, and PerplexityBot, measured per domain. 16.8% of 10000 domains block at least one AI crawler.

Canonical: https://crawlwise.site/reports/ai-crawler-access
Last updated: 2026-11-04

Original data · AI crawler access

16.8% of 10000 domains block at least one AI crawler.

Sample of 10000 domains, collected 2026-11-04.

## Headline

| Sample | 10000 domains |
| --- | --- |
| Blocking at least one AI crawler | 1679 (16.8%) |
| Blocking a retrieval crawler | 1543 (15.4%) |
| Challenged rather than blocked | 360 (3.6%) |

## By crawler

| Crawler | Allowed | Blocked | Challenged | Not established |
| --- | --- | --- | --- | --- |
| Bytespider | 1634 | 1538 | 322 | 6506 |
| Amazonbot | 1740 | 1546 | 226 | 6488 |
| GPTBot | 1745 | 1545 | 211 | 6499 |
| ClaudeBot | 1753 | 1527 | 216 | 6504 |
| CCBot | 1773 | 1526 | 213 | 6488 |
| Meta-ExternalAgent | 1791 | 1493 | 221 | 6495 |
| Diffbot | 1797 | 1497 | 216 | 6490 |
| cohere-ai | 1804 | 1504 | 203 | 6489 |
| PerplexityBot | 1825 | 1470 | 218 | 6487 |
| Timpibot | 1828 | 1460 | 227 | 6485 |
| ChatGPT-User | 1816 | 1466 | 220 | 6498 |
| OAI-SearchBot | 1843 | 1444 | 223 | 6490 |
| Perplexity-User | 1837 | 1443 | 224 | 6496 |
| Claude-SearchBot | 1858 | 1434 | 226 | 6482 |
| Claude-User | 1863 | 1432 | 226 | 6479 |
| Google-Extended | 3565 | 802 | 0 | 5633 |
| Applebot-Extended | 3603 | 764 | 0 | 5633 |

## By industry

| Industry | Domains | Blocking anything |
| --- | --- | --- |
| unclassified | 10000 | 1679 (16.8%) |

## The shared AI-training opt-out

On 30 October 2026 Google, Microsoft, OpenAI and Anthropic backed one robots.txt group that states an AI-training preference to every participating provider: `User-agent: __usage_ai-training` followed by `Disallow: /`. It comes from an [IETF AIPREF draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/). Google aims to honour it in 2027 and Microsoft from February 2027. It states a preference about training; it does not block crawling, and `User-agent: *` does not count as using it.

**4 of 4367 domains (0.1%)** use it to opt out. A domain counts only when its robots.txt was read.

Most popular first: twitter.com, x.com, imdb.com, www.imdb.com

Add it with the free [robots.txt generator](https://crawlwise.site/tools/robots-txt) (template “Shared AI-training opt-out”), and check any site with the [AI crawler checker](https://crawlwise.site/tools/ai-crawler-check).

## Check your own site

Run the free [AI crawler checker](https://crawlwise.site/tools/ai-crawler-check) on any URL to see which of these crawlers can read it, from robots.txt and a live fetch.

## What changed since 1 October 2026

Robots.txt policy only, on the 4282 domains whose robots.txt was read in both the 1 October 2026 and 4 November 2026 runs. **37 domains** added or removed a rule for at least one AI crawler. A live fetch that answered differently is not counted: that is usually a CDN, not a decision.

**Do not compare the headline shares.** The 1 October 2026 run counted a 404 or 5xx answer to a crawler as a block. Since 26 October 2026 that is recorded as an error, not a block, so the share of domains blocking a crawler is lower now for that reason alone. The robots.txt comparison below is not affected.

| Crawler | Disallowed before | Disallowed now | Added a block | Removed a block |
| --- | --- | --- | --- | --- |
| Google-Extended | 767 | 782 | 22 | 7 |
| GPTBot | 783 | 800 | 22 | 5 |
| ClaudeBot | 758 | 775 | 22 | 5 |
| Applebot-Extended | 728 | 745 | 22 | 5 |
| Amazonbot | 745 | 764 | 23 | 4 |
| Meta-ExternalAgent | 759 | 768 | 15 | 6 |
| PerplexityBot | 747 | 756 | 14 | 5 |
| CCBot | 796 | 803 | 13 | 6 |
| Bytespider | 786 | 794 | 13 | 5 |
| cohere-ai | 760 | 767 | 12 | 5 |
| Diffbot | 760 | 767 | 12 | 5 |
| Claude-SearchBot | 734 | 737 | 8 | 5 |
| OAI-SearchBot | 736 | 736 | 5 | 5 |
| Claude-User | 741 | 741 | 5 | 5 |
| Perplexity-User | 742 | 742 | 5 | 5 |
| ChatGPT-User | 751 | 750 | 4 | 5 |
| Timpibot | 742 | 742 | 4 | 4 |

**Newly blocking Google-Extended:** facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.

**No longer blocking Google-Extended:** juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, cnn.com, infojobs.net, www.cnn.com.

**Newly blocking GPTBot:** facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.

**No longer blocking GPTBot:** juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, infojobs.net.

**Newly blocking ClaudeBot:** facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.

**No longer blocking ClaudeBot:** juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, infojobs.net.

**Newly blocking Applebot-Extended:** facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.

**No longer blocking Applebot-Extended:** juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, infojobs.net.

**Newly blocking Amazonbot:** facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 13 more.

**No longer blocking Amazonbot:** juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com.

## Methodology, in full

**Sample.** This run probed 10000 domains from the Cisco Umbrella top-one-million list (public download, rank order 1 through 10000). Industry is not in that list; domains without a hand label are grouped as unclassified. That is a popularity-ranked sample, not a random sample of the web and not an exhaustive census, so every share below describes this list and not the whole internet. A domain that fails to resolve is recorded as not established rather than dropped, because dropping failures is how a sample silently becomes favourable.

**robots.txt.** Fetched from the origin, verified to be a robots document rather than an HTML challenge page, then resolved with an RFC 9309 matcher against each crawler’s product token and the probed path. A file that could not be read as a robots file leaves that crawler not established.

**Live fetch.** The URL is fetched with that crawler’s user-agent. The response is classified by status, by challenge fingerprints (Cloudflare, Akamai, DataDome), and by body size against an ordinary fetch. A 401, a 403 or a challenge is a block. A 404 or a 5xx is the page answering an error, and a 429 or a timeout is the server being busy or slow: all three are not established, not a block.

**Verdicts.** Allowed, blocked, challenged, or not established. A disallow in robots.txt for a training crawler is recorded as a deliberate opt-out, and it is counted separately from an accidental block everywhere on this page.

**What this cannot tell you.** One probe from one location at one moment: a CDN answering a crawler differently in another region is invisible here. A crawler that ignores robots.txt is not stopped by it, and this measures policy as much as enforcement. Rate limiting is only called a block when the response is a 403. And a domain blocking a training crawler while allowing the retrieval crawlers from the same owner is reported as exactly that, not as one number.

## Crawlers probed

| Crawler | Owner | Category | User agent sent |
| --- | --- | --- | --- |
| GPTBot | OpenAI | training | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot` |
| OAI-SearchBot | OpenAI | retrieval | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot` |
| ChatGPT-User | OpenAI | retrieval | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot` |
| ClaudeBot | Anthropic | training | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)` |
| Claude-SearchBot | Anthropic | retrieval | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)` |
| Claude-User | Anthropic | retrieval | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)` |
| Google-Extended | Google | training | `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` |
| PerplexityBot | Perplexity | retrieval | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)` |
| Perplexity-User | Perplexity | retrieval | `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)` |
| CCBot | Common Crawl | research | `CCBot/2.0 (https://commoncrawl.org/faq/)` |
| Bytespider | ByteDance | training | `Mozilla/5.0 (compatible; Bytespider; +https://zhanzhang.toutiao.com/)` |
| Applebot-Extended | Apple | training | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)` |
| Meta-ExternalAgent | Meta | training | `Mozilla/5.0 (compatible; Meta-ExternalAgent/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)` |
| Amazonbot | Amazon | training | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)` |
| cohere-ai | Cohere | retrieval | `Mozilla/5.0 (compatible; cohere-ai/1.0; +https://cohere.com/bot)` |
| Diffbot | Diffbot | research | `Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)` |
| Timpibot | Timpi | research | `Mozilla/5.0 (compatible; Timpibot/0.1; +https://timpi.io)` |

## Reproduce it

```
node scripts/fetch-crawler-report-tranco.mjs --limit 10000
node scripts/crawler-report.mjs --list scripts/fixtures/crawler-report-tranco.csv --limit 10000 --label top-10000 --concurrency 4
```

A larger sample is the same command with a longer list, at roughly nineteen outbound requests per domain. 10,000 domains is about 190,000 requests and several hours; the page states whichever sample size it is reading, and nothing here is extrapolated from a smaller one.

[Check one page yourself, free](https://crawlwise.site/tools/ai-crawler-check)