robots.txt generator
The robots.txt generator writes crawl rules for search engines and AI crawlers such as GPTBot and ClaudeBot. You decide which bots to allow; it keeps training opt-outs separate from search access.
- Free, no sign-up
- Runs in your browser
- Generator · 35 free tools
- No Sitemap declared. Add the absolute URL of your sitemap.xml.
User-agent: *
Allow: /Crawler reference
| User-agent | Owner | Category |
|---|---|---|
__usage_ai-training | Shared (IETF AIPREF) | training |
Googlebot | search | |
Googlebot-Image | search | |
Google-Extended | training | |
AdsBot-Google | search | |
Bingbot | Microsoft | search |
MicrosoftPreview | Microsoft | social |
DuckDuckBot | DuckDuckGo | search |
DuckAssistBot | DuckDuckGo | retrieval |
Applebot | Apple | search |
Applebot-Extended | Apple | training |
GPTBot | OpenAI | training |
OAI-SearchBot | OpenAI | retrieval |
ChatGPT-User | OpenAI | retrieval |
ClaudeBot | Anthropic | training |
anthropic-ai | Anthropic | training |
Claude-SearchBot | Anthropic | retrieval |
Claude-User | Anthropic | retrieval |
PerplexityBot | Perplexity | retrieval |
Perplexity-User | Perplexity | retrieval |
Google-CloudVertexBot | retrieval | |
GoogleOther | research | |
Meta-ExternalAgent | Meta | training |
Meta-ExternalFetcher | Meta | retrieval |
facebookexternalhit | Meta | social |
Bytespider | ByteDance | training |
CCBot | Common Crawl | research |
Diffbot | Diffbot | research |
Cohere-ai | Cohere | retrieval |
cohere-training-data-crawler | Cohere | training |
MistralAI-User | Mistral | retrieval |
Amazonbot | Amazon | training |
YandexBot | Yandex | search |
Baiduspider | Baidu | search |
Twitterbot | X | social |
LinkedInBot | social | |
Slackbot | Slack | social |
Discordbot | Discord | social |
WhatsApp | Meta | social |
AhrefsBot | Ahrefs | seo |
SemrushBot | Semrush | seo |
DotBot | Moz | seo |
Once the file is live, check that the page actually serves it, and that nothing else on the page undoes it. Plans start at $14.99 a month.
How to use the robots.txt generator
- 1
Fill in the fields
Or press Load example to see a finished version you can edit.
- 2
Watch the output
The file updates as you type, and missing required fields are flagged.
- 3
Copy or download
Use the buttons on the output box. Nothing is sent to us unless you press Save.
- 4
Add it to your site
Paste it where the guide says, then check it with the matching checker.
What this does not measure
We would rather say what a tool cannot see than let a clean result mislead you.
- robots.txt is a request, not enforcement. A crawler that ignores it is not stopped by it.
- It does not know which paths exist on your site. Ask the crawl for that.
- Blocking a training crawler and blocking a search crawler are different decisions with different consequences, and the generator does not make either one for you.
Want the full picture?
This tool answers one question. A full audit runs every check on a page, scores it, and gives you a short list of what to fix first.
Questions about the robots.txt generator
More in the guides.
Keep reading
robots.txt for AI crawlers: copy-paste examples with explanations
Three robots.txt files you can paste today, and why each line is there.
Does robots.txt control AI crawlers? What it does and does not do
robots.txt is a request, not a lock. Here is where it works for AI crawlers and where it does not.
Which AI crawlers should you allow? A vendor-by-vendor decision guide
Blocking training and blocking answer visibility are different decisions. Here is each vendor’s split.