# robots.txt generator

> Free robots.txt generator. Write crawl rules for Googlebot, Bingbot, and AI crawlers like GPTBot and ClaudeBot, and keep training opt-outs separate. No sign-up.

Canonical: https://crawlwise.site/tools/robots-txt

The robots.txt generator writes crawl rules for search engines and AI crawlers such as GPTBot and ClaudeBot. You decide which bots to allow; it keeps training opt-outs separate from search access.

- Free, no sign-up
- Runs in your browser
- Generator · 35 free tools

- No Sitemap declared. Add the absolute URL of your sitemap.xml.

*robots.txt*

```
User-agent: *
Allow: /
```

### Crawler reference

| User-agent | Owner | Category |
| --- | --- | --- |
| `__usage_ai-training` | Shared (IETF AIPREF) | training |
| `Googlebot` | Google | search |
| `Googlebot-Image` | Google | search |
| `Google-Extended` | Google | training |
| `AdsBot-Google` | Google | search |
| `Bingbot` | Microsoft | search |
| `MicrosoftPreview` | Microsoft | social |
| `DuckDuckBot` | DuckDuckGo | search |
| `DuckAssistBot` | DuckDuckGo | retrieval |
| `Applebot` | Apple | search |
| `Applebot-Extended` | Apple | training |
| `GPTBot` | OpenAI | training |
| `OAI-SearchBot` | OpenAI | retrieval |
| `ChatGPT-User` | OpenAI | retrieval |
| `ClaudeBot` | Anthropic | training |
| `anthropic-ai` | Anthropic | training |
| `Claude-SearchBot` | Anthropic | retrieval |
| `Claude-User` | Anthropic | retrieval |
| `PerplexityBot` | Perplexity | retrieval |
| `Perplexity-User` | Perplexity | retrieval |
| `Google-CloudVertexBot` | Google | retrieval |
| `GoogleOther` | Google | research |
| `Meta-ExternalAgent` | Meta | training |
| `Meta-ExternalFetcher` | Meta | retrieval |
| `facebookexternalhit` | Meta | social |
| `Bytespider` | ByteDance | training |
| `CCBot` | Common Crawl | research |
| `Diffbot` | Diffbot | research |
| `Cohere-ai` | Cohere | retrieval |
| `cohere-training-data-crawler` | Cohere | training |
| `MistralAI-User` | Mistral | retrieval |
| `Amazonbot` | Amazon | training |
| `YandexBot` | Yandex | search |
| `Baiduspider` | Baidu | search |
| `Twitterbot` | X | social |
| `LinkedInBot` | LinkedIn | social |
| `Slackbot` | Slack | social |
| `Discordbot` | Discord | social |
| `WhatsApp` | Meta | social |
| `AhrefsBot` | Ahrefs | seo |
| `SemrushBot` | Semrush | seo |
| `DotBot` | Moz | seo |

Once the file is live, check that the page actually serves it, and that nothing else on the page undoes it. Plans start at $14.99 a month.

[Audit the live page](https://crawlwise.site/#audit-url) [See plans](https://crawlwise.site/plans)

## How to use the robots.txt generator

1. ### Fill in the fields

   Or press Load example to see a finished version you can edit.
2. ### Watch the output

   The file updates as you type, and missing required fields are flagged.
3. ### Copy or download

   Use the buttons on the output box. Nothing is sent to us unless you press Save.
4. ### Add it to your site

   Paste it where the guide says, then check it with the matching checker.

### What this does not measure

We would rather say what a tool cannot see than let a clean result mislead you.

- robots.txt is a request, not enforcement. A crawler that ignores it is not stopped by it.
- It does not know which paths exist on your site. Ask the crawl for that.
- Blocking a training crawler and blocking a search crawler are different decisions with different consequences, and the generator does not make either one for you.

Want the full picture?

This tool answers one question. A full audit runs every check on a page, scores it, and gives you a short list of what to fix first.

[Run a free audit](https://crawlwise.site/#audit-url)

## Questions about the robots.txt generator

More in the [guides](https://crawlwise.site/guides).

### What does the robots.txt generator do?

Write crawl rules for search and retrieval bots. Training preferences are a separate decision.

### Do I need a Crawlwise account to use robots.txt generator?

No. Every tool in the free toolkit runs without sign-up, card, or email. A full site audit and saved history require an account and credits.

### What can robots.txt generator not tell me?

robots.txt is a request, not enforcement. A crawler that ignores it is not stopped by it.

### How is this different from a paid SEO suite?

Crawlwise shows the evidence behind each result and prices work in credits. Suites bundle backlinks and ads research; we focus on measurable on-page health and crawler access.

### Can I run the same check on my whole site?

The free tool answers one URL or one question. Add a site in Crawlwise to crawl many pages, export reports, and track changes over time on paid plans.

### Will fixing what the tool flags improve rankings?

Nothing outside Google can promise rankings. Fixing real technical issues removes friction for crawlers and readers; it is necessary but not sufficient for visibility.

## Keep reading

[Blog](https://crawlwise.site/blog) [All guides](https://crawlwise.site/guides)

- [robots.txt for AI crawlers: copy-paste examples with explanations](https://crawlwise.site/guides/ai-search/robots-txt-for-ai-crawlers-examples): Three robots.txt files you can paste today, and why each line is there.
- [Does robots.txt control AI crawlers? What it does and does not do](https://crawlwise.site/guides/ai-search/does-robots-txt-control-ai-crawlers): robots.txt is a request, not a lock. Here is where it works for AI crawlers and where it does not.
- [Which AI crawlers should you allow? A vendor-by-vendor decision guide](https://crawlwise.site/guides/ai-search/which-ai-crawlers-to-allow): Blocking training and blocking answer visibility are different decisions. Here is each vendor’s split.