Does fwmrm.net block AI crawlers?
fwmrm.net serves a bot challenge to 17, and allows 0 (checked 2026-10-01).
The blocks are on research and corpus crawlers rather than on the assistants people use to search, so citation in ChatGPT, Claude and Perplexity is not affected by them.
Every crawler we tested
| Crawler | Used for | Verdict | What we saw |
|---|---|---|---|
| GPTBot OpenAI | ChatGPT (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| OAI-SearchBot OpenAI | ChatGPT search (retrieval) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| ChatGPT-User OpenAI | ChatGPT (retrieval) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| ClaudeBot Anthropic | Claude (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Claude-SearchBot Anthropic | Claude search (retrieval) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Claude-User Anthropic | Claude (retrieval) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Google-Extended Google | Gemini (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| PerplexityBot Perplexity | Perplexity (retrieval) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Perplexity-User Perplexity | Perplexity (retrieval) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| CCBot Common Crawl | Common Crawl (research) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Bytespider ByteDance | ByteDance (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Applebot-Extended Apple | Apple Intelligence (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Meta-ExternalAgent Meta | Meta AI (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Amazonbot Amazon | Alexa (training) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| cohere-ai Cohere | Cohere (retrieval) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Diffbot Diffbot | Diffbot (research) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
| Timpibot Timpi | Timpi (research) | Challenged | Challenged: robots.txt permits it, but the fetch got a bot challenge page instead of the content. |
How this was measured
We read fwmrm.net’s robots.txt, resolved it with an RFC 9309 matcher for each crawler’s product token, then fetched the homepage once as each crawler. A 403 or a challenge page is a block even when robots.txt allows the crawler; a timeout or rate limit is “not established”, never a block. One probe, from one location, on 2026-10-01, so a rule changed since then is not reflected here. Full methodology.
Check your own site
The free checker runs the same probe on any URL: robots.txt resolved for every AI crawler, then a live fetch as each one, so a CDN or firewall block shows up even when robots.txt looks fine.
Need it across every page? Site crawls start at $4.99 a month, and Pro adds monitoring that re-checks your URLs on a schedule.
Other sites
- jquery.com — blocks none of 17
- ipify.org — blocks none of 17
- giphy.com — blocks none of 17
- qualtrics.com — blocks none of 17
- bugsnag.com — blocks none of 17
- pinimg.com — blocks 16 of 17