# What is CCBot?

> CCBot (Common Crawl): what it does, how it helps your site show up in AI products, and how to test access on your own URL.

Canonical: https://crawlwise.site/learn/crawler/ccbot

AI crawler · research · Common Crawl

Common Crawl's bot. It builds an open web archive that many research and training pipelines read indirectly.

### How this helps you show up

Letting CCBot through spreads your content into a widely used corpus. That can matter if you want your version of the facts to exist in datasets others train on — a long-game visibility play, not a day-one traffic lever.

### If it cannot reach your page

Future Common Crawl snapshots skip you. Models already trained on older snapshots do not forget you overnight — but new training runs might never see your latest pages.

### The facts

**robots.txt token**

CCBot

**Owner**

Common Crawl

**Product it serves**

Common Crawl

**Category**

research — an open corpus or knowledge graph

**Training opt-out**

Documented — only affects training-style use, not every AI product

**User agent as sent**

`CCBot/2.0 (https://commoncrawl.org/faq/)`

robots.txt is a request, not a lock. Our checker fetches your page as CCBot and reports what actually came back — status code, challenge page, or real HTML.

### Check your site in a minute

The [AI crawler checker](https://crawlwise.site/tools/ai-crawler-check) tests CCBot plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the [robots.txt tester](https://crawlwise.site/tools/robots-tester).

### Other AI crawlers

- [GPTBot](https://crawlwise.site/learn/crawler/gptbot) — OpenAI
- [OAI-SearchBot](https://crawlwise.site/learn/crawler/oai-searchbot) — OpenAI
- [ChatGPT-User](https://crawlwise.site/learn/crawler/chatgpt-user) — OpenAI
- [ClaudeBot](https://crawlwise.site/learn/crawler/claudebot) — Anthropic
- [Claude-SearchBot](https://crawlwise.site/learn/crawler/claude-searchbot) — Anthropic
- [Claude-User](https://crawlwise.site/learn/crawler/claude-user) — Anthropic
- [Google-Extended](https://crawlwise.site/learn/crawler/google-extended) — Google
- [PerplexityBot](https://crawlwise.site/learn/crawler/perplexitybot) — Perplexity
- [Perplexity-User](https://crawlwise.site/learn/crawler/perplexity-user) — Perplexity
- [Bytespider](https://crawlwise.site/learn/crawler/bytespider) — ByteDance
- [Applebot-Extended](https://crawlwise.site/learn/crawler/applebot-extended) — Apple
- [Meta-ExternalAgent](https://crawlwise.site/learn/crawler/meta-externalagent) — Meta
- [Amazonbot](https://crawlwise.site/learn/crawler/amazonbot) — Amazon
- [cohere-ai](https://crawlwise.site/learn/crawler/cohere-ai) — Cohere
- [Diffbot](https://crawlwise.site/learn/crawler/diffbot) — Diffbot
- [Timpibot](https://crawlwise.site/learn/crawler/timpibot) — Timpi