# AI crawlers and AI search, answered plainly

> What each AI crawler does, how to stay visible in ChatGPT, Claude, Perplexity, and more, with honest comparisons and free tools to test your own URLs.

Canonical: https://crawlwise.site/learn

Learn · no sign-up

Plain-language guides so you can show up in AI answers — what each crawler does, which ones to keep reachable for citations, and free tools to verify your own site.

## AI crawlers

[GPTBot](https://crawlwise.site/learn/crawler/gptbot): OpenAI · training

[OAI-SearchBot](https://crawlwise.site/learn/crawler/oai-searchbot): OpenAI · retrieval

[ChatGPT-User](https://crawlwise.site/learn/crawler/chatgpt-user): OpenAI · retrieval

[ClaudeBot](https://crawlwise.site/learn/crawler/claudebot): Anthropic · training

[Claude-SearchBot](https://crawlwise.site/learn/crawler/claude-searchbot): Anthropic · retrieval

[Claude-User](https://crawlwise.site/learn/crawler/claude-user): Anthropic · retrieval

[Google-Extended](https://crawlwise.site/learn/crawler/google-extended): Google · training

[PerplexityBot](https://crawlwise.site/learn/crawler/perplexitybot): Perplexity · retrieval

[Perplexity-User](https://crawlwise.site/learn/crawler/perplexity-user): Perplexity · retrieval

[CCBot](https://crawlwise.site/learn/crawler/ccbot): Common Crawl · research

[Bytespider](https://crawlwise.site/learn/crawler/bytespider): ByteDance · training

[Applebot-Extended](https://crawlwise.site/learn/crawler/applebot-extended): Apple · training

[Meta-ExternalAgent](https://crawlwise.site/learn/crawler/meta-externalagent): Meta · training

[Amazonbot](https://crawlwise.site/learn/crawler/amazonbot): Amazon · training

[cohere-ai](https://crawlwise.site/learn/crawler/cohere-ai): Cohere · retrieval

[Diffbot](https://crawlwise.site/learn/crawler/diffbot): Diffbot · research

[Timpibot](https://crawlwise.site/learn/crawler/timpibot): Timpi · research

## Comparisons

[vs Screaming Frog SEO Spider](https://crawlwise.site/vs/screaming-frog): The desktop crawler most technical SEOs already have open.

[vs Ahrefs Site Audit](https://crawlwise.site/vs/ahrefs-site-audit): An audit bolted to the largest independent backlink index.

[vs Semrush Site Audit](https://crawlwise.site/vs/semrush-site-audit): An enterprise SEO suite with an audit attached to a very large keyword database.

[vs open-seo](https://crawlwise.site/vs/open-seo): An open-source DataForSEO interface you can self-host.

[vs Sitebulb](https://crawlwise.site/vs/sitebulb): A desktop crawler built around prioritised hints and visual reports.

[vs Seobility](https://crawlwise.site/vs/seobility): A cloud SEO suite with a site audit, rank tracking and backlinks in one subscription.

[vs SEO Site Checkup](https://crawlwise.site/vs/seo-site-checkup): A long-running instant checkup that has grown into a monitoring suite.

[vs squirrelscan](https://crawlwise.site/vs/squirrelscan): An open-source website QA CLI built to hand its findings to a coding agent.

[vs SE Ranking](https://crawlwise.site/vs/se-ranking): An all-in-one SEO platform with rank tracking, a large audit allowance and AI search tracking.

[vs Ubersuggest](https://crawlwise.site/vs/ubersuggest): A keyword research and SEO suite sold with a one-time lifetime option.

[vs Moz Pro](https://crawlwise.site/vs/moz): A long-standing SEO suite known for Domain Authority and its link index.

[vs Lighthouse and PageSpeed Insights](https://crawlwise.site/vs/lighthouse): Google’s free, open-source page auditor, also run inside PageSpeed Insights.

[vs Google Search Console](https://crawlwise.site/vs/google-search-console): Google’s free service for search traffic data and indexing problems.

## Blog & guides

[Blog](https://crawlwise.site/blog): Workflows, crawler access, and audit stories — with FAQs and comparison tables.

[Guides](https://crawlwise.site/guides): Playbooks for every free tool, plans, and site visibility — step by step.

## The free toolkit

[All free tools](https://crawlwise.site/tools): Twenty-seven tools, no account, no credit, no email.

[What crawlwise cannot tell you](https://crawlwise.site/limits): The limits page. The one page here that is not selling you anything.