robots.txt tester: a calm step-by-step playbook

You picked a specific job, and that is the best way to start. Does this URL get blocked, for this crawler? Test the real file against a real path. Free, no sign-up. This playbook is the calm version of that job: when to open robots.txt tester, how to read what comes back, and when a full Crawlwise audit is the kinder next step. Nothing here is a ranking promise. Everything here is something you can check again after you fix it.

When this is the right tool

Reach for robots.txt tester when you already have a page, a file, or a question in front of you. People usually arrive while they are:

• robots.txt tester
• is this url blocked by robots.txt
• robots.txt validator

If you wanted a backlink graph, a thousand tracked keywords, or a crawl of a 50,000-page catalogue, this is the wrong drawer. Those products exist, and they are good at that work. This one is good at the page you can point at today.

A five-minute sitting

1. Open the tool. No account, no card, no email.
2. Paste the URL or the values the form asks for. Use the live page, not a memory of it.
3. Read the evidence beside the result. A raw value you can argue with is more useful than a lonely pass or fail.
4. Fix it on staging, run the same check again, and ship when the second run matches what you meant.

That second run is the part people skip, and it is the part that makes the afternoon feel finished. You get to see the page agree with you.

A good check is a conversation with the page. robots.txt tester starts that conversation. You get to finish it.

Adnan Arodiya, Crawlwise

What we will say out loud

Every tool on Crawlwise has a fence around it. These are the fences for robots.txt tester, written so you can trust the rest of the page:

• It implements RFC 9309, which is the specification. Crawlers mostly follow it and a few do not, and this tool cannot tell you which of those you are dealing with from the outside.
• It only evaluates the file as served today. A CDN or WAF rule that answers differently for a bot is a separate question, and the AI crawler checker is the tool for that.
• It does not check that your sitemap is reachable, only that the directive is present.

When a check cannot run, we would rather say so than invent a number. Unknown is a real answer. It is also a kinder one than a fake zero on a report you might send to a client.

What to do with a clean result

A clean result is permission to move on, not a trophy. Pair robots.txt tester with the neighbours that catch a different kind of miss: meta tags for what the page claims, crawler access for who can read it, and Core Web Vitals for how it feels to load. Most launch bugs are a small stack of those, not one dramatic failure.

When the page is about to go live, or a client is about to ask “is this ready?”, run a full audit. You get the checks that apply, a plan tied to evidence, and a place to save the story once you have an account. History is how next month’s you thanks this month’s you.

What a plan actually costs

The free tools stay free. You can open them, run them, and leave without an account. A plan is for the moment you want the report saved, a few sites watched, or the same checks from the API. The price on this table is the price at checkout. A yearly plan is ten months of the monthly price, so two months are on us.

A single-page audit is about 1.25 credits, and that includes the live probe of answer-engine crawlers. You see the estimate before anything runs. If a hold is not used, it comes back to your balance. Extra credits, when you already subscribe, are $9.99 for 20.

You have enough to begin

Open robots.txt tester, run it on the URL you care about, and fix the first true thing it shows you. Then run it again. That loop is the whole craft, and you are already in it. If you want the wider picture, the plans above are sized for one site, a few sites, or a client list. Pick the smallest one that will not run out mid-month, and leave the rest for later.