Which websites block AI crawlers?

62 of the 250 popular sites listed here block at least one AI crawler, measured 2026-10-01.

Each site links to its own page: every crawler we tested, what its robots.txt says, and what a live fetch got back. The list is drawn from the 10,000-domain run, keeping sites whose robots.txt we could read and whose homepage answered, in popularity order.

By crawler

• Sites that block GPTBot — 38 of 239
• Sites that block OAI-SearchBot — 26 of 243
• Sites that block ChatGPT-User — 30 of 240
• Sites that block ClaudeBot — 36 of 240
• Sites that block Claude-SearchBot — 23 of 244
• Sites that block Claude-User — 24 of 243
• Sites that block Google-Extended — 38 of 244
• Sites that block PerplexityBot — 29 of 242
• Sites that block Perplexity-User — 27 of 243
• Sites that block CCBot — 37 of 240
• Sites that block Bytespider — 44 of 238
• Sites that block Applebot-Extended — 27 of 244
• Sites that block Meta-ExternalAgent — 36 of 241
• Sites that block Amazonbot — 31 of 237
• Sites that block cohere-ai — 31 of 244
• Sites that block Diffbot — 34 of 243
• Sites that block Timpibot — 29 of 246

All sites

Check your own site

The free checker runs the same probe on any URL: robots.txt resolved for every AI crawler, then a live fetch as each one, so a CDN or firewall block shows up even when robots.txt looks fine.

Check AI crawler access, free

Need it across every page? Site crawls start at $4.99 a month, and Pro adds monitoring that re-checks your URLs on a schedule.