Check · files · scored
Googlebot access in robots.txt: what it measures
Whether robots.txt lets Googlebot crawl this exact URL, using the Googlebot group and the longest matching rule (RFC 9309). A robots.txt that answers a server error counts as a block, because Google stops crawling until it recovers. A block holds the score at 40.
How it is determined
The check reads the response your server returns and records what it found. That observation is stored with the report, so reopening an old audit shows what was true on the day it ran rather than what the page says now. It is not a live measurement, and it is not a model’s opinion of your page.
Ruleset robots-googlebot · category files · can count towards the checklist score
What it cannot tell you
- Whether Google has indexed the page. Search Console's URL Inspection says that.
- Whether a firewall or CDN refuses Googlebot even though robots.txt allows it.
If it failed, what to do
Remove or narrow the rule that blocks the URL, unless you mean to keep it out of Google.
Then verify it yourself rather than trusting the next report: reload the page, view the source, and look at the thing the check described. A check you can falsify in ten seconds is the only kind worth having.
Settle it without an account
The free robots tester checks this on one URL and needs no signup — the same rule, on the same page, with no credit spent. When you want the whole page assessed, an audit is covered by the trial credit.