Learn · no sign-up
Answers to the questions people actually ask
Plain-language guides so you can show up in AI answers — what each crawler does, which ones to keep reachable for citations, and free tools to verify your own site.
AI crawlers
GPTBot
OpenAI · training
OAI-SearchBot
OpenAI · retrieval
ChatGPT-User
OpenAI · retrieval
ClaudeBot
Anthropic · training
Claude-SearchBot
Anthropic · retrieval
Claude-User
Anthropic · retrieval
Google-Extended
Google · training
PerplexityBot
Perplexity · retrieval
Perplexity-User
Perplexity · retrieval
CCBot
Common Crawl · research
Bytespider
ByteDance · training
Applebot-Extended
Apple · training
Meta-ExternalAgent
Meta · training
Amazonbot
Amazon · training
cohere-ai
Cohere · retrieval
Diffbot
Diffbot · research
Timpibot
Timpi · research
Comparisons
vs Screaming Frog SEO Spider
The desktop crawler most technical SEOs already have open.
vs Ahrefs Site Audit
An audit bolted to the largest independent backlink index.
vs Semrush Site Audit
An enterprise SEO suite with an audit attached to a very large keyword database.
vs open-seo
An open-source DataForSEO interface you can self-host.