# What is Diffbot?

> Diffbot (Diffbot): what it does, how it helps your site show up in AI products, and how to test access on your own URL.

Canonical: https://crawlwise.site/learn/crawler/diffbot

AI crawler · research · Diffbot

A commercial crawler that extracts structured facts for a knowledge graph sold to other products.

### How this helps you show up

Useful when you want your official specs, pricing, or entity data to be the version third-party tools reuse — especially for B2B or data-heavy sites.

### If it cannot reach your page

Your pages stop updating in their graph; widgets and integrations built on Diffbot may show stale or competitor data.

### The facts

**robots.txt token**

Diffbot

**Owner**

Diffbot

**Product it serves**

Diffbot

**Category**

research — an open corpus or knowledge graph

**Training opt-out**

Documented — only affects training-style use, not every AI product

**User agent as sent**

`Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)`

robots.txt is a request, not a lock. Our checker fetches your page as Diffbot and reports what actually came back — status code, challenge page, or real HTML.

### Check your site in a minute

The [AI crawler checker](https://crawlwise.site/tools/ai-crawler-check) tests Diffbot plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the [robots.txt tester](https://crawlwise.site/tools/robots-tester).

### Other AI crawlers

- [GPTBot](https://crawlwise.site/learn/crawler/gptbot) — OpenAI
- [OAI-SearchBot](https://crawlwise.site/learn/crawler/oai-searchbot) — OpenAI
- [ChatGPT-User](https://crawlwise.site/learn/crawler/chatgpt-user) — OpenAI
- [ClaudeBot](https://crawlwise.site/learn/crawler/claudebot) — Anthropic
- [Claude-SearchBot](https://crawlwise.site/learn/crawler/claude-searchbot) — Anthropic
- [Claude-User](https://crawlwise.site/learn/crawler/claude-user) — Anthropic
- [Google-Extended](https://crawlwise.site/learn/crawler/google-extended) — Google
- [PerplexityBot](https://crawlwise.site/learn/crawler/perplexitybot) — Perplexity
- [Perplexity-User](https://crawlwise.site/learn/crawler/perplexity-user) — Perplexity
- [CCBot](https://crawlwise.site/learn/crawler/ccbot) — Common Crawl
- [Bytespider](https://crawlwise.site/learn/crawler/bytespider) — ByteDance
- [Applebot-Extended](https://crawlwise.site/learn/crawler/applebot-extended) — Apple
- [Meta-ExternalAgent](https://crawlwise.site/learn/crawler/meta-externalagent) — Meta
- [Amazonbot](https://crawlwise.site/learn/crawler/amazonbot) — Amazon
- [cohere-ai](https://crawlwise.site/learn/crawler/cohere-ai) — Cohere
- [Timpibot](https://crawlwise.site/learn/crawler/timpibot) — Timpi