# What is Google-Extended?

> Google-Extended (Google): what it does, how it helps your site show up in AI products, and how to test access on your own URL.

Canonical: https://crawlwise.site/learn/crawler/google-extended

AI crawler · training · Google

A robots.txt token for whether Google may use your content in Gemini and Vertex training — not the same thing as Googlebot for Search.

### How this helps you show up

Most sites that care about Google visibility leave Search crawling alone and think separately about training. Allowing Google-Extended does not replace good Search SEO; it is an extra lane into how Gemini learns about your niche.

### If it cannot reach your page

You opt out of Gemini training uses covered by this token. Google Search rankings are a different system — blocking Google-Extended does not protect or hurt them by itself.

### The facts

**robots.txt token**

Google-Extended

**Owner**

Google

**Product it serves**

Gemini

**Category**

training — building model training data

**Training opt-out**

Documented — only affects training-style use, not every AI product

**User agent as sent**

`Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)`

robots.txt is a request, not a lock. Our checker fetches your page as Google-Extended and reports what actually came back — status code, challenge page, or real HTML.

### Check your site in a minute

The [AI crawler checker](https://crawlwise.site/tools/ai-crawler-check) tests Google-Extended plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the [robots.txt tester](https://crawlwise.site/tools/robots-tester).

### Other AI crawlers

- [GPTBot](https://crawlwise.site/learn/crawler/gptbot) — OpenAI
- [OAI-SearchBot](https://crawlwise.site/learn/crawler/oai-searchbot) — OpenAI
- [ChatGPT-User](https://crawlwise.site/learn/crawler/chatgpt-user) — OpenAI
- [ClaudeBot](https://crawlwise.site/learn/crawler/claudebot) — Anthropic
- [Claude-SearchBot](https://crawlwise.site/learn/crawler/claude-searchbot) — Anthropic
- [Claude-User](https://crawlwise.site/learn/crawler/claude-user) — Anthropic
- [PerplexityBot](https://crawlwise.site/learn/crawler/perplexitybot) — Perplexity
- [Perplexity-User](https://crawlwise.site/learn/crawler/perplexity-user) — Perplexity
- [CCBot](https://crawlwise.site/learn/crawler/ccbot) — Common Crawl
- [Bytespider](https://crawlwise.site/learn/crawler/bytespider) — ByteDance
- [Applebot-Extended](https://crawlwise.site/learn/crawler/applebot-extended) — Apple
- [Meta-ExternalAgent](https://crawlwise.site/learn/crawler/meta-externalagent) — Meta
- [Amazonbot](https://crawlwise.site/learn/crawler/amazonbot) — Amazon
- [cohere-ai](https://crawlwise.site/learn/crawler/cohere-ai) — Cohere
- [Diffbot](https://crawlwise.site/learn/crawler/diffbot) — Diffbot
- [Timpibot](https://crawlwise.site/learn/crawler/timpibot) — Timpi