# Five crawler-blocking mistakes that hide your best pages > Robots.txt surprises, CDN rules, and staging leaks — common ways sites block retrieval bots without knowing it. Canonical: https://crawlwise.site/blog/answer-engine-crawlers/answer-engine-crawler-blocking-mistakes Last updated: 2026-03-18 Blocking training bots is not the same as blocking retrieval. Here is how to tell what you blocked. by Adnan Arodiya Updated March 18, 2026 · 8 min read ## Key takeaways - Copy-paste robots rules often block more than you intend. - Test with live fetches, not assumptions from your robots file. - Staging noindex leaks can poison production signals. - Document which bots you allow for search vs retrieval. Teams block crawlers for good reasons — scrapers, training data worries, cost control. The failure mode is blocking retrieval bots that power answers about your brand. ## Mistake 1: Disallow everything, then allow Google only Many templates disallow `/` for all agents except a short allow list. Retrieval bots may not be on that list. ## Mistake 2: Using staging rules in production Basic-auth or IP allowlists left enabled after launch return 401/403 to bots. ## Mistake 3: Confusing training with search Blocking GPTBot does not automatically block ChatGPT-User or other retrieval agents. Test each name you care about. ## Mistake 4: CDN bot scores Aggressive bot management can throttle legitimate fetchers. Whitelist after you identify false positives. ## Mistake 5: Never re-testing Ship a fix, then [re-audit](https://crawlwise.site/) or re-run the [crawler check](https://crawlwise.site/tools/ai-crawler-check) on the same URL. ## You are already further along than it feels “Five crawler-blocking mistakes that hide your best pages” is the kind of page people open when something feels off and they want a calm next step. That instinct is the whole job. You do not need a new stack, a new agency, or a weekend of tabs. You need one true thing you can fix, then another, then a re-check so you can see the page get cleaner. > You do not need a perfect site this week. You need the next true fix, and the patience to check it again after you ship. > > Adnan Arodiya, Crawlwise Start with the page a customer actually lands on. Read what the server returns, not what you remember pasting into the CMS. If a crawler is blocked, fix that before you rewrite a headline. If a title is doing its job, leave it alone and move to the next check. Small honest wins beat a dashboard full of red. When the free check has told you what it can, a full audit is the natural next sitting. It scores the checks that apply, says when something was not measured, and keeps the story if you save it. Plans start at $14.99 a month for Starter. Growth and Agency are there when a few sites become a client list. Nothing here asks you to buy a seat you will not use. ## What a plan actually costs The free tools stay free. You can open them, run them, and leave without an account. A plan is for the moment you want the report saved, a few sites watched, or the same checks from the API. The price on this table is the price at checkout. A yearly plan is ten months of the monthly price, so two months are on us. | Plan | Monthly | Credits | A good fit when | | --- | --- | --- | --- | | Starter | $14.99 | 40 | Regular audits of up to three of your own sites, with Search Console and watched URLs | | Growth | $49.99 | 150 | Client work: backlinks, keyword research, rank tracking and the API | | Agency | $99.99 | 400 | Many locations, reported under your own name | A single-page audit is about 1.25 credits, and that includes the live probe of answer-engine crawlers. You see the estimate before anything runs. If a hold is not used, it comes back to your balance. Extra credits, when you already subscribe, are $9.99 for 20. ## A comparison worth keeping A typical SEO suite is wonderful at the work it was built for: a giant keyword database, a backlink index, ads research, seats for a team. Crawlwise is wonderful at a smaller, sharper job. We show the evidence on the page, we test whether answer-engine crawlers can actually fetch it, and we price the work in credits you can see before you click. If you need their index, keep it. If you need a clear read on this URL, you are in the right place. Take the win that is in front of you. Run the check, fix one thing on staging, and come back. The page will still be here, and so will we. ## Crawlwise vs a typical SEO suite | What you get | Crawlwise | Typical suite | | --- | --- | --- | | Monthly price for audits + crawler checks | From $14.99 | $139+ typical suite | | Evidence shown per finding | Value + check name | Often score only | | Answer-engine crawler live test | Included in audit | Rare | | Fractional credit pricing | Yes | Seat bundles | | Backlink index | Not our focus | Core | Try it on your URL ## Run the checks on your live site Free tools answer one question. A full audit scores the page, tests answer-engine crawlers, and saves to history on a plan. - Live crawler probes - Core Web Vitals - One free full audit [Get my free report](https://crawlwise.site/#audit-url) [Compare plans](https://crawlwise.site/plans) ## Frequently asked questions ### Should I block all bots I do not recognize? No. Blocking retrieval bots can remove your pages from answers that cite live fetches. Block deliberately, with a written policy. ### Does robots.txt enforce blocking? It is a request. Well-behaved bots honor it; attackers do not. Always verify with a live crawler test. ### What is the difference between training and retrieval crawlers? Training crawlers ingest content for model building; retrieval crawlers fetch pages to answer user queries. Policies differ — conflating them causes mistakes. ### Can Cloudflare rules block crawlers? Yes. WAF, bot fight mode, and country blocks can return 403 to legitimate retrieval bots. Check response codes per bot. ### Will allowing crawlers hurt my Google rankings? Googlebot is separate from answer-engine retrieval bots. Each should be tested on its own; blocking one does not block the other. ### How often should I re-test? After every deploy that touches robots, CDN, or auth middleware — and monthly on key landing pages. ### Where do I learn per-crawler behavior? See the crawler library under /learn or run the free checker on /tools/ai-crawler-check. ### Does Crawlwise fix robots for me? We report what each bot received. You edit robots or CDN rules; re-test until fetches succeed. ## Meet the author Adnan Arodiya Crawlwise Writes about what Crawlwise actually measures: on-page evidence, crawler access, and performance signals — with the limits stated up front.