Check · files · scored
robots.txt discovery: what it measures
Whether /robots.txt answers and whether its content parses as a robots file.
How it is determined
The check reads the response your server returns and records what it found. That observation is stored with the report, so reopening an old audit shows what was true on the day it ran rather than what the page says now. It is not a live measurement, and it is not a model’s opinion of your page.
Ruleset robots · category files · can count towards the checklist score
What it cannot tell you
- Whether the rules inside it are the ones you want. Use the robots tester for a specific URL and crawler.
- Whether a CDN or WAF answers differently for a bot.
If it failed, what to do
Serve a robots.txt that parses, and point it at your sitemap.
Then verify it yourself rather than trusting the next report: reload the page, view the source, and look at the thing the check described. A check you can falsify in ten seconds is the only kind worth having.
Settle it without an account
The free robots tester checks this on one URL and needs no signup — the same rule, on the same page, with no credit spent. When you want the whole page assessed, an audit is covered by the trial credit.