Skip to content
crawlwise.

Search Crawlwise

Search tools, articles, guides and your sites

robots.txt tester

The robots.txt tester fetches a site's live robots.txt and tells you whether a given URL is allowed or blocked for a given crawler, and which rule decided it.

  • Free, no sign-up
  • Results in seconds
  • Checker · 35 free tools

This checks one page. The same problem usually repeats on every page built from the same template. Plans start at $14.99 a month.

Crawl the whole site Run a full audit

How to use the robots.txt tester

  1. 1

    Paste an address

    A full URL or just a domain. We add https:// when it is missing.

  2. 2

    We fetch it live

    Our server requests the page now, the way a crawler would, and records what came back.

  3. 3

    Read the evidence

    Every result shows the value we read, so you can see why it passed or failed.

  4. 4

    Fix and check again

    Change the page, run it once more, and ship when the second run agrees.

What this does not measure

We would rather say what a tool cannot see than let a clean result mislead you.

  • It implements RFC 9309, which is the specification. Crawlers mostly follow it and a few do not, and this tool cannot tell you which of those you are dealing with from the outside.
  • It only evaluates the file as served today. A CDN or WAF rule that answers differently for a bot is a separate question, and the AI crawler checker is the tool for that.
  • It does not check that your sitemap is reachable, only that the directive is present.

Want the full picture?

This tool answers one question. A full audit runs every check on a page, scores it, and gives you a short list of what to fix first.

Run a free audit

Questions about the robots.txt tester

More in the guides.

Does this URL get blocked, for this crawler? Test the real file against a real path. Free, no sign-up.

No. Every tool in the free toolkit runs without sign-up, card, or email. A full site audit and saved history require an account and credits.

It implements RFC 9309, which is the specification. Crawlers mostly follow it and a few do not, and this tool cannot tell you which of those you are dealing with from the outside.

Crawlwise shows the evidence behind each result and prices work in credits. Suites bundle backlinks and ads research; we focus on measurable on-page health and crawler access.

The free tool answers one URL or one question. Add a site in Crawlwise to crawl many pages, export reports, and track changes over time on paid plans.

Nothing outside Google can promise rankings. Fixing real technical issues removes friction for crawlers and readers; it is necessary but not sufficient for visibility.

Keep reading

Blog All guides