Learn · no sign-up
AI crawlers and AI search, answered plainly
Plain-language guides so you can show up in AI answers — what each crawler does, which ones to keep reachable for citations, and free tools to verify your own site.
AI crawlers
GPTBot
OpenAI · training
OAI-SearchBot
OpenAI · retrieval
ChatGPT-User
OpenAI · retrieval
ClaudeBot
Anthropic · training
Claude-SearchBot
Anthropic · retrieval
Claude-User
Anthropic · retrieval
Google-Extended
Google · training
PerplexityBot
Perplexity · retrieval
Perplexity-User
Perplexity · retrieval
CCBot
Common Crawl · research
Bytespider
ByteDance · training
Applebot-Extended
Apple · training
Meta-ExternalAgent
Meta · training
Amazonbot
Amazon · training
cohere-ai
Cohere · retrieval
Diffbot
Diffbot · research
Timpibot
Timpi · research
Comparisons
vs Screaming Frog SEO Spider
The desktop crawler most technical SEOs already have open.
vs Ahrefs Site Audit
An audit bolted to the largest independent backlink index.
vs Semrush Site Audit
An enterprise SEO suite with an audit attached to a very large keyword database.
vs open-seo
An open-source DataForSEO interface you can self-host.
vs Sitebulb
A desktop crawler built around prioritised hints and visual reports.
vs Seobility
A cloud SEO suite with a site audit, rank tracking and backlinks in one subscription.
vs SEO Site Checkup
A long-running instant checkup that has grown into a monitoring suite.
vs squirrelscan
An open-source website QA CLI built to hand its findings to a coding agent.
vs SE Ranking
An all-in-one SEO platform with rank tracking, a large audit allowance and AI search tracking.
vs Ubersuggest
A keyword research and SEO suite sold with a one-time lifetime option.
vs Moz Pro
A long-standing SEO suite known for Domain Authority and its link index.
vs Lighthouse and PageSpeed Insights
Google’s free, open-source page auditor, also run inside PageSpeed Insights.
vs Google Search Console
Google’s free service for search traffic data and indexing problems.