Does meraki.com block AI crawlers?
meraki.com blocks 15 of 17 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, PerplexityBot, Perplexity-User, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, Diffbot and Timpibot) and serves a bot challenge to 2, and allows 0 (checked 2026-10-01).
Retrieval crawlers are blocked too, so AI assistants that fetch a page live to quote and link it cannot read this site. Its content can still appear in answers from older training data or third-party copies, but not as a fresh, cited source.
Every crawler we tested
| Crawler | Used for | Verdict | What we saw |
|---|---|---|---|
| GPTBot OpenAI | ChatGPT (training) | Blocked | Blocked: robots.txt disallows it. |
| OAI-SearchBot OpenAI | ChatGPT search (retrieval) | Blocked | Blocked: robots.txt disallows it. |
| ChatGPT-User OpenAI | ChatGPT (retrieval) | Blocked | Blocked: robots.txt disallows it. |
| ClaudeBot Anthropic | Claude (training) | Blocked | Blocked: robots.txt disallows it. |
| Claude-SearchBot Anthropic | Claude search (retrieval) | Blocked | Blocked: robots.txt disallows it. |
| Claude-User Anthropic | Claude (retrieval) | Blocked | Blocked: robots.txt disallows it. |
| Google-Extended Google | Gemini (training) | Blocked | Blocked: robots.txt disallows it. |
| PerplexityBot Perplexity | Perplexity (retrieval) | Blocked | Blocked: robots.txt disallows it. |
| Perplexity-User Perplexity | Perplexity (retrieval) | Blocked | Blocked: robots.txt disallows it. |
| CCBot Common Crawl | Common Crawl (research) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Bytespider ByteDance | ByteDance (training) | Blocked | Blocked: robots.txt disallows it. |
| Applebot-Extended Apple | Apple Intelligence (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Meta-ExternalAgent Meta | Meta AI (training) | Blocked | Blocked: robots.txt disallows it. |
| Amazonbot Amazon | Alexa (training) | Blocked | Blocked: robots.txt disallows it. |
| cohere-ai Cohere | Cohere (retrieval) | Blocked | Blocked: robots.txt disallows it. |
| Diffbot Diffbot | Diffbot (research) | Blocked | Blocked: robots.txt disallows it. |
| Timpibot Timpi | Timpi (research) | Blocked | Blocked: robots.txt disallows it. |
How that compares
- GPTBot is blocked by 38 of the 239 sites on these pages (15.9%).
- OAI-SearchBot is blocked by 26 of the 243 sites on these pages (10.7%).
- ChatGPT-User is blocked by 30 of the 240 sites on these pages (12.5%).
- ClaudeBot is blocked by 36 of the 240 sites on these pages (15%).
How this was measured
We read meraki.com’s robots.txt, resolved it with an RFC 9309 matcher for each crawler’s product token, then fetched the homepage once as each crawler. A 403 or a challenge page is a block even when robots.txt allows the crawler; a timeout or rate limit is “not established”, never a block. One probe, from one location, on 2026-10-01, so a rule changed since then is not reflected here. Full methodology.
Check your own site
The free checker runs the same probe on any URL: robots.txt resolved for every AI crawler, then a live fetch as each one, so a CDN or firewall block shows up even when robots.txt looks fine.
Need it across every page? Site crawls start at $4.99 a month, and Pro adds monitoring that re-checks your URLs on a schedule.
Other sites
- amazonvideo.com — blocks 10 of 17
- quantummetric.com — blocks none of 17
- identrust.com — blocks none of 17
- slack.com — blocks none of 17
- fullstory.com — blocks none of 17
- mountain.com — blocks none of 17