AI crawler · research · Diffbot
What is Diffbot?
A commercial crawler that extracts structured facts for a knowledge graph sold to other products.
How this helps you show up
Useful when you want your official specs, pricing, or entity data to be the version third-party tools reuse — especially for B2B or data-heavy sites.
If it cannot reach your page
Your pages stop updating in their graph; widgets and integrations built on Diffbot may show stale or competitor data.
The facts
- robots.txt token
- Diffbot
- Owner
- Diffbot
- Product it serves
- Diffbot
- Category
- research — an open corpus or knowledge graph
- Training opt-out
- Documented — only affects training-style use, not every AI product
- User agent as sent
Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
robots.txt is a request, not a lock. Our checker fetches your page as Diffbot and reports what actually came back — status code, challenge page, or real HTML.
Check your site in a minute
The AI crawler checker tests Diffbot plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the robots.txt tester.
Other AI crawlers
- GPTBot — OpenAI
- OAI-SearchBot — OpenAI
- ChatGPT-User — OpenAI
- ClaudeBot — Anthropic
- Claude-SearchBot — Anthropic
- Claude-User — Anthropic
- Google-Extended — Google
- PerplexityBot — Perplexity
- Perplexity-User — Perplexity
- CCBot — Common Crawl
- Bytespider — ByteDance
- Applebot-Extended — Apple
- Meta-ExternalAgent — Meta
- Amazonbot — Amazon
- cohere-ai — Cohere
- Timpibot — Timpi