Skip to content

analyzer · free · no sign-up

robots.txt tester

Does this URL get blocked, for this crawler? Test the real file against a real path. Free, no sign-up.

Answers: robots.txt tester · is this url blocked by robots.txt · robots.txt validator

Read the step-by-step guide · Blog workflow

What this does not measure

  • It implements RFC 9309, which is the specification. Crawlers mostly follow it and a few do not, and this tool cannot tell you which of those you are dealing with from the outside.
  • It only evaluates the file as served today. A CDN or WAF rule that answers differently for a bot is a separate question, and the AI crawler checker is the tool for that.
  • It does not check that your sitemap is reachable, only that the directive is present.

Evaluates the real robots.txt against a real path using the RFC 9309 matcher. A robots.txt rule is a request, not enforcement — the AI crawler checker is the tool that fetches the page as the crawler and reports what came back.