# robots.txt tester > Free robots.txt tester. Check whether a URL is blocked for Googlebot, GPTBot, or any user agent, against the live robots.txt, and see the matching rule. Canonical: https://crawlwise.site/tools/robots-tester The robots.txt tester fetches a site's live robots.txt and tells you whether a given URL is allowed or blocked for a given crawler, and which rule decided it. - Free, no sign-up - Results in seconds - Checker ยท 35 free tools This checks one page. The same problem usually repeats on every page built from the same template. Plans start at $14.99 a month. [Crawl the whole site](https://crawlwise.site/plans) [Run a full audit](https://crawlwise.site/#audit-url) ## How to use the robots.txt tester 1. ### Paste an address A full URL or just a domain. We add https:// when it is missing. 2. ### We fetch it live Our server requests the page now, the way a crawler would, and records what came back. 3. ### Read the evidence Every result shows the value we read, so you can see why it passed or failed. 4. ### Fix and check again Change the page, run it once more, and ship when the second run agrees. ### What this does not measure We would rather say what a tool cannot see than let a clean result mislead you. - It implements RFC 9309, which is the specification. Crawlers mostly follow it and a few do not, and this tool cannot tell you which of those you are dealing with from the outside. - It only evaluates the file as served today. A CDN or WAF rule that answers differently for a bot is a separate question, and the AI crawler checker is the tool for that. - It does not check that your sitemap is reachable, only that the directive is present. Want the full picture? This tool answers one question. A full audit runs every check on a page, scores it, and gives you a short list of what to fix first. [Run a free audit](https://crawlwise.site/#audit-url) ## Questions about the robots.txt tester More in the [guides](https://crawlwise.site/guides). ### What does the robots.txt tester do? Does this URL get blocked, for this crawler? Test the real file against a real path. Free, no sign-up. ### Do I need a Crawlwise account to use robots.txt tester? No. Every tool in the free toolkit runs without sign-up, card, or email. A full site audit and saved history require an account and credits. ### What can robots.txt tester not tell me? It implements RFC 9309, which is the specification. Crawlers mostly follow it and a few do not, and this tool cannot tell you which of those you are dealing with from the outside. ### How is this different from a paid SEO suite? Crawlwise shows the evidence behind each result and prices work in credits. Suites bundle backlinks and ads research; we focus on measurable on-page health and crawler access. ### Can I run the same check on my whole site? The free tool answers one URL or one question. Add a site in Crawlwise to crawl many pages, export reports, and track changes over time on paid plans. ### Will fixing what the tool flags improve rankings? Nothing outside Google can promise rankings. Fixing real technical issues removes friction for crawlers and readers; it is necessary but not sufficient for visibility. ## Keep reading [Blog](https://crawlwise.site/blog) [All guides](https://crawlwise.site/guides) - [robots.txt in the wild: what the top 10,000 domains actually serve](https://crawlwise.site/blog/technical-seo/robots-txt-top-10000-domains): Original data: only 1,837 of 10,000 popular domains served a robots.txt a crawler could read as one. - [noindex vs robots.txt disallow: which one removes a page from Google?](https://crawlwise.site/blog/technical-seo/noindex-vs-robots-txt-disallow): Disallow stops crawling. noindex stops indexing. Using both at once means Google never reads the noindex. - [Does robots.txt control AI crawlers? What it does and does not do](https://crawlwise.site/guides/ai-search/does-robots-txt-control-ai-crawlers): robots.txt is a request, not a lock. Here is where it works for AI crawlers and where it does not.