Skip to content
crawlwise.

AI crawler · training · Google

What is Google-Extended?

A robots.txt token for whether Google may use your content in Gemini and Vertex training — not the same thing as Googlebot for Search.

How this helps you show up

Most sites that care about Google visibility leave Search crawling alone and think separately about training. Allowing Google-Extended does not replace good Search SEO; it is an extra lane into how Gemini learns about your niche.

If it cannot reach your page

You opt out of Gemini training uses covered by this token. Google Search rankings are a different system — blocking Google-Extended does not protect or hurt them by itself.

The facts

robots.txt token
Google-Extended
Owner
Google
Product it serves
Gemini
Category
training — building model training data
Training opt-out
Documented — only affects training-style use, not every AI product
User agent as sent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

robots.txt is a request, not a lock. Our checker fetches your page as Google-Extended and reports what actually came back — status code, challenge page, or real HTML.

Check your site in a minute

The AI crawler checker tests Google-Extended plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the robots.txt tester.

Other AI crawlers