generator · free · no sign-up
robots.txt generator
Write crawl rules for search and retrieval bots. Training preferences are a separate decision.
Answers: robots.txt generator · block gptbot robots.txt · disallow all robots
What this does not measure
- robots.txt is a request, not enforcement. A crawler that ignores it is not stopped by it.
- It does not know which paths exist on your site. Ask the crawl for that.
- Blocking a training crawler and blocking a search crawler are different decisions with different consequences, and the generator does not make either one for you.
- No Sitemap declared. Add the absolute URL of your sitemap.xml.
User-agent: *Crawler reference
| User-agent | Owner | Category |
|---|---|---|
Googlebot | search | |
Googlebot-Image | search | |
Google-Extended | training | |
AdsBot-Google | search | |
Bingbot | Microsoft | search |
MicrosoftPreview | Microsoft | retrieval |
DuckDuckBot | DuckDuckGo | search |
Applebot | Apple | search |
Applebot-Extended | Apple | training |
GPTBot | OpenAI | training |
OAI-SearchBot | OpenAI | retrieval |
ChatGPT-User | OpenAI | retrieval |
ClaudeBot | Anthropic | training |
Claude-SearchBot | Anthropic | retrieval |
Claude-User | Anthropic | retrieval |
PerplexityBot | Perplexity | retrieval |
Perplexity-User | Perplexity | retrieval |
Google-CloudVertexBot | training | |
Meta-ExternalAgent | Meta | training |
Meta-ExternalFetcher | Meta | retrieval |
facebookexternalhit | Meta | social |
Bytespider | ByteDance | training |
CCBot | Common Crawl | research |
Diffbot | Diffbot | research |
Cohere-ai | Cohere | retrieval |
cohere-training-data-crawler | Cohere | training |
Mistral | Mistral | training |
Grok | xAI | retrieval |
Amazonbot | Amazon | training |
YandexBot | Yandex | search |
Baiduspider | Baidu | search |
Twitterbot | X | social |
LinkedInBot | social | |
Slackbot | Slack | social |
Discordbot | Discord | social |
WhatsApp | Meta | social |
AhrefsBot | Ahrefs | seo |
SemrushBot | Semrush | seo |
DotBot | Moz | seo |