Skip to content
crawlwise.

AI crawler · retrieval · Anthropic

What is Claude-SearchBot?

Anthropic's retrieval crawler for Claude search — it finds pages that Claude can quote or link to.

How this helps you show up

Keep this one happy. Readable HTML, real text on the page (not only in images), and a robots.txt that allows Claude-SearchBot are the boring basics that get you cited.

If it cannot reach your page

Claude search will not quote a page its crawler cannot load. You stay invisible in that product even if your Google SEO is fine.

The facts

robots.txt token
Claude-SearchBot
Owner
Anthropic
Product it serves
Claude search
Category
retrieval — indexed so an answer can cite it
Training opt-out
Retrieval / user fetch — usually keep open for citations
User agent as sent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)

robots.txt is a request, not a lock. Our checker fetches your page as Claude-SearchBot and reports what actually came back — status code, challenge page, or real HTML.

Check your site in a minute

The AI crawler checker tests Claude-SearchBot plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the robots.txt tester.

Other AI crawlers