AI crawler · retrieval · Anthropic
What is Claude-User?
A fetch when someone asks Claude about your site or pastes your link.
How this helps you show up
Same idea as ChatGPT-User: a real person is looking at your brand right now. If the fetch works, your words can shape the reply.
If it cannot reach your page
Claude answers without your page in the mix.
The facts
- robots.txt token
- Claude-User
- Owner
- Anthropic
- Product it serves
- Claude
- Category
- retrieval — indexed so an answer can cite it
- Training opt-out
- Retrieval / user fetch — usually keep open for citations
- User agent as sent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
robots.txt is a request, not a lock. Our checker fetches your page as Claude-User and reports what actually came back — status code, challenge page, or real HTML.
Check your site in a minute
The AI crawler checker tests Claude-User plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the robots.txt tester.
Other AI crawlers
- GPTBot — OpenAI
- OAI-SearchBot — OpenAI
- ChatGPT-User — OpenAI
- ClaudeBot — Anthropic
- Claude-SearchBot — Anthropic
- Google-Extended — Google
- PerplexityBot — Perplexity
- Perplexity-User — Perplexity
- CCBot — Common Crawl
- Bytespider — ByteDance
- Applebot-Extended — Apple
- Meta-ExternalAgent — Meta
- Amazonbot — Amazon
- cohere-ai — Cohere
- Diffbot — Diffbot
- Timpibot — Timpi