Five crawler-blocking mistakes that hide your best pages

Teams block crawlers for good reasons — scrapers, training data worries, cost control. The failure mode is blocking retrieval bots that power answers about your brand.

Mistake 1: Disallow everything, then allow Google only

Many templates disallow / for all agents except a short allow list. Retrieval bots may not be on that list.

Mistake 2: Using staging rules in production

Basic-auth or IP allowlists left enabled after launch return 401/403 to bots.

Mistake 3: Confusing training with search

Blocking GPTBot does not automatically block ChatGPT-User or other retrieval agents. Test each name you care about.

Mistake 4: CDN bot scores

Aggressive bot management can throttle legitimate fetchers. Whitelist after you identify false positives.

Mistake 5: Never re-testing

Ship a fix, then re-audit or re-run the crawler check on the same URL.

You are already further along than it feels

“Five crawler-blocking mistakes that hide your best pages” is the kind of page people open when something feels off and they want a calm next step. That instinct is the whole job. You do not need a new stack, a new agency, or a weekend of tabs. You need one true thing you can fix, then another, then a re-check so you can see the page get cleaner.

You do not need a perfect site this week. You need the next true fix, and the patience to check it again after you ship.

Adnan Arodiya, Crawlwise

Start with the page a customer actually lands on. Read what the server returns, not what you remember pasting into the CMS. If a crawler is blocked, fix that before you rewrite a headline. If a title is doing its job, leave it alone and move to the next check. Small honest wins beat a dashboard full of red.

When the free check has told you what it can, a full audit is the natural next sitting. It scores the checks that apply, says when something was not measured, and keeps the story if you save it. Plans start at $4.99 a month for Starter. Pro, Studio, and Agency are there when one site becomes a few, or a few become a client list. Nothing here asks you to buy a seat you will not use.

What a plan actually costs

The free tools stay free. You can open them, run them, and leave without an account. A plan is for the moment you want the report saved, a few sites watched, or the same checks from the API. The price on this table is the price at checkout. A yearly plan is ten months of the monthly price, so two months are on us.

A single-page audit is about 1.25 credits, and that includes the live probe of answer-engine crawlers. You see the estimate before anything runs. If a hold is not used, it comes back to your balance. Extra credits, when you already subscribe, are $9.99 for 20.

A comparison worth keeping

A typical SEO suite is wonderful at the work it was built for: a giant keyword database, a backlink index, ads research, seats for a team. Crawlwise is wonderful at a smaller, sharper job. We show the evidence on the page, we test whether answer-engine crawlers can actually fetch it, and we price the work in credits you can see before you click. If you need their index, keep it. If you need a clear read on this URL, you are in the right place.

Take the win that is in front of you. Run the check, fix one thing on staging, and come back. The page will still be here, and so will we.