Skip to content
crawlwise.

AI crawler · training · Anthropic

What is ClaudeBot?

Anthropic's training crawler — it gathers pages for model training, not for the live citation you see in Claude search.

How this helps you show up

Allowing it does not hurt classic search rankings. Claude's quotable answers lean on Claude-SearchBot and Claude-User. If you want Claude to mention you in search, those are the names to keep open.

If it cannot reach your page

You opt out of training only. Claude can still answer questions using pages its retrieval bots already know — unless those are blocked too.

The facts

robots.txt token
ClaudeBot
Owner
Anthropic
Product it serves
Claude
Category
training — building model training data
Training opt-out
Documented — only affects training-style use, not every AI product
User agent as sent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

robots.txt is a request, not a lock. Our checker fetches your page as ClaudeBot and reports what actually came back — status code, challenge page, or real HTML.

Check your site in a minute

The AI crawler checker tests ClaudeBot plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the robots.txt tester.

Other AI crawlers