AI crawler statistics: the AI Crawler Access Report
16.8% of 10000 domains block at least one AI crawler.
Sample of 10000 domains, collected 2026-11-04.
Headline
| Sample | 10000 domains |
|---|---|
| Blocking at least one AI crawler | 1679 (16.8%) |
| Blocking a retrieval crawler | 1543 (15.4%) |
| Challenged rather than blocked | 360 (3.6%) |
By crawler
| Crawler | Allowed | Blocked | Challenged | Not established |
|---|---|---|---|---|
| Bytespider | 1634 | 1538 | 322 | 6506 |
| Amazonbot | 1740 | 1546 | 226 | 6488 |
| GPTBot | 1745 | 1545 | 211 | 6499 |
| ClaudeBot | 1753 | 1527 | 216 | 6504 |
| CCBot | 1773 | 1526 | 213 | 6488 |
| Meta-ExternalAgent | 1791 | 1493 | 221 | 6495 |
| Diffbot | 1797 | 1497 | 216 | 6490 |
| cohere-ai | 1804 | 1504 | 203 | 6489 |
| PerplexityBot | 1825 | 1470 | 218 | 6487 |
| Timpibot | 1828 | 1460 | 227 | 6485 |
| ChatGPT-User | 1816 | 1466 | 220 | 6498 |
| OAI-SearchBot | 1843 | 1444 | 223 | 6490 |
| Perplexity-User | 1837 | 1443 | 224 | 6496 |
| Claude-SearchBot | 1858 | 1434 | 226 | 6482 |
| Claude-User | 1863 | 1432 | 226 | 6479 |
| Google-Extended | 3565 | 802 | 0 | 5633 |
| Applebot-Extended | 3603 | 764 | 0 | 5633 |
By industry
| Industry | Domains | Blocking anything |
|---|---|---|
| unclassified | 10000 | 1679 (16.8%) |
The shared AI-training opt-out
On 30 October 2026 Google, Microsoft, OpenAI and Anthropic backed one robots.txt group that states an AI-training preference to every participating provider: User-agent: __usage_ai-training followed by Disallow: /. It comes from an IETF AIPREF draft. Google aims to honour it in 2027 and Microsoft from February 2027. It states a preference about training; it does not block crawling, and User-agent: * does not count as using it.
4 of 4367 domains (0.1%) use it to opt out. A domain counts only when its robots.txt was read.
Most popular first: twitter.com, x.com, imdb.com, www.imdb.com
Add it with the free robots.txt generator (template “Shared AI-training opt-out”), and check any site with the AI crawler checker.
Check your own site
Run the free AI crawler checker on any URL to see which of these crawlers can read it, from robots.txt and a live fetch.
What changed since 1 October 2026
Robots.txt policy only, on the 4282 domains whose robots.txt was read in both the 1 October 2026 and 4 November 2026 runs. 37 domains added or removed a rule for at least one AI crawler. A live fetch that answered differently is not counted: that is usually a CDN, not a decision.
Do not compare the headline shares. The 1 October 2026 run counted a 404 or 5xx answer to a crawler as a block. Since 26 October 2026 that is recorded as an error, not a block, so the share of domains blocking a crawler is lower now for that reason alone. The robots.txt comparison below is not affected.
| Crawler | Disallowed before | Disallowed now | Added a block | Removed a block |
|---|---|---|---|---|
| Google-Extended | 767 | 782 | 22 | 7 |
| GPTBot | 783 | 800 | 22 | 5 |
| ClaudeBot | 758 | 775 | 22 | 5 |
| Applebot-Extended | 728 | 745 | 22 | 5 |
| Amazonbot | 745 | 764 | 23 | 4 |
| Meta-ExternalAgent | 759 | 768 | 15 | 6 |
| PerplexityBot | 747 | 756 | 14 | 5 |
| CCBot | 796 | 803 | 13 | 6 |
| Bytespider | 786 | 794 | 13 | 5 |
| cohere-ai | 760 | 767 | 12 | 5 |
| Diffbot | 760 | 767 | 12 | 5 |
| Claude-SearchBot | 734 | 737 | 8 | 5 |
| OAI-SearchBot | 736 | 736 | 5 | 5 |
| Claude-User | 741 | 741 | 5 | 5 |
| Perplexity-User | 742 | 742 | 5 | 5 |
| ChatGPT-User | 751 | 750 | 4 | 5 |
| Timpibot | 742 | 742 | 4 | 4 |
Newly blocking Google-Extended: facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.
No longer blocking Google-Extended: juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, cnn.com, infojobs.net, www.cnn.com.
Newly blocking GPTBot: facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.
No longer blocking GPTBot: juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, infojobs.net.
Newly blocking ClaudeBot: facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.
No longer blocking ClaudeBot: juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, infojobs.net.
Newly blocking Applebot-Extended: facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 12 more.
No longer blocking Applebot-Extended: juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com, infojobs.net.
Newly blocking Amazonbot: facebook.com, www.facebook.com, web.facebook.com, lookaside.facebook.com, ep7.facebook.com, usync.4dex.io, m.facebook.com, mobile.facebook.com, oaistatsig.com, mfx.mobilefuse.com and 13 more.
No longer blocking Amazonbot: juicyscore.com, track.bidgx.com, srv.bidgx.com, a.magsrv.com.
Methodology, in full
Sample. This run probed 10000 domains from the Cisco Umbrella top-one-million list (public download, rank order 1 through 10000). Industry is not in that list; domains without a hand label are grouped as unclassified. That is a popularity-ranked sample, not a random sample of the web and not an exhaustive census, so every share below describes this list and not the whole internet. A domain that fails to resolve is recorded as not established rather than dropped, because dropping failures is how a sample silently becomes favourable.
robots.txt. Fetched from the origin, verified to be a robots document rather than an HTML challenge page, then resolved with an RFC 9309 matcher against each crawler’s product token and the probed path. A file that could not be read as a robots file leaves that crawler not established.
Live fetch. The URL is fetched with that crawler’s user-agent. The response is classified by status, by challenge fingerprints (Cloudflare, Akamai, DataDome), and by body size against an ordinary fetch. A 401, a 403 or a challenge is a block. A 404 or a 5xx is the page answering an error, and a 429 or a timeout is the server being busy or slow: all three are not established, not a block.
Verdicts. Allowed, blocked, challenged, or not established. A disallow in robots.txt for a training crawler is recorded as a deliberate opt-out, and it is counted separately from an accidental block everywhere on this page.
What this cannot tell you. One probe from one location at one moment: a CDN answering a crawler differently in another region is invisible here. A crawler that ignores robots.txt is not stopped by it, and this measures policy as much as enforcement. Rate limiting is only called a block when the response is a 403. And a domain blocking a training crawler while allowing the retrieval crawlers from the same owner is reported as exactly that, not as one number.
Crawlers probed
| Crawler | Owner | Category | User agent sent |
|---|---|---|---|
| GPTBot | OpenAI | training | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot |
| OAI-SearchBot | OpenAI | retrieval | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot |
| ChatGPT-User | OpenAI | retrieval | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
| ClaudeBot | Anthropic | training | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
| Claude-SearchBot | Anthropic | retrieval | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com) |
| Claude-User | Anthropic | retrieval | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) |
| Google-Extended | training | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| PerplexityBot | Perplexity | retrieval | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Perplexity-User | Perplexity | retrieval | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
| CCBot | Common Crawl | research | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Bytespider | ByteDance | training | Mozilla/5.0 (compatible; Bytespider; +https://zhanzhang.toutiao.com/) |
| Applebot-Extended | Apple | training | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| Meta-ExternalAgent | Meta | training | Mozilla/5.0 (compatible; Meta-ExternalAgent/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Amazonbot | Amazon | training | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) |
| cohere-ai | Cohere | retrieval | Mozilla/5.0 (compatible; cohere-ai/1.0; +https://cohere.com/bot) |
| Diffbot | Diffbot | research | Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com) |
| Timpibot | Timpi | research | Mozilla/5.0 (compatible; Timpibot/0.1; +https://timpi.io) |
Reproduce it
node scripts/fetch-crawler-report-tranco.mjs --limit 10000 node scripts/crawler-report.mjs --list scripts/fixtures/crawler-report-tranco.csv --limit 10000 --label top-10000 --concurrency 4
A larger sample is the same command with a longer list, at roughly nineteen outbound requests per domain. 10,000 domains is about 190,000 requests and several hours; the page states whichever sample size it is reading, and nothing here is extrapolated from a smaller one.