Webflow SEO audit: robots.txt, AI bot settings and what to check

Webflow gives you a clean place for every crawl setting, which makes it easy to forget that nothing changes until you publish, and that a staging subdomain exists alongside your real one. Audit the published result, not the settings panel.

Where are the crawler settings?

Open Site settings and go to SEO. That is where you write a custom robots.txt, control indexing of the webflow.io subdomain, and choose whether the sitemap is listed in robots.txt. Webflow’s help center also documents controls for AI bots in the same area, including toggles that generate robots.txt rules for groups of known bots, and Webflow has announced work with Cloudflare to give site owners more control over AI crawlers.

How do you allow or block one AI crawler?

Add a group to the custom robots.txt field and publish:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

That keeps OpenAI’s training crawler out while letting the crawler behind ChatGPT search fetch your pages to cite them. If you also use a bot toggle, open /robots.txt after publishing and check the two did not produce contradictory groups: a crawler obeys only the most specific group that names it. Does robots.txt control AI crawlers? explains how groups are matched.

Nothing in Webflow settings is real until it is published. Check the live file, then check it again from a crawler’s point of view.

Adnan Arodiya, Crawlwise

What else belongs in a Webflow audit?

• The staging subdomain. Make sure the webflow.io address is set not to be indexed, so it does not show up as a duplicate of your domain.
• CMS templates. Collection pages share one template, so bind the title, meta description and Open Graph fields to collection fields instead of leaving one static value for every item.
• Per-page indexing. Pages excluded with the page-level indexing option should also be left out of the sitemap.
• Interactions and embeds. Content that only appears after an interaction or a third-party embed loads may not be in the server HTML a crawler reads.

Then submit the sitemap in Search Console and Bing Webmaster Tools, and re-check after every significant publish.

Check it on your own Webflow site

Run the free AI crawler checker on your homepage and one product or article page. It reads your robots.txt the way each crawler does, then fetches the page as GPTBot, ClaudeBot, PerplexityBot and 14 others, so a block added by a CDN, firewall or app shows up even when the file looks right. The robots.txt tester shows which rule decides a given path, and a full audit adds titles, canonicals, structured data, hreflang and performance for the same URL.

To see how popular sites set these rules, look any of them up in the site-by-site AI crawler results.

What a plan actually costs

The free tools stay free. You can open them, run them, and leave without an account. A plan is for the moment you want the report saved, a few sites watched, or the same checks from the API. The price on this table is the price at checkout. A yearly plan is ten months of the monthly price, so two months are on us.

A single-page audit is about 1.25 credits, and that includes the live probe of answer-engine crawlers. You see the estimate before anything runs. If a hold is not used, it comes back to your balance. Extra credits, when you already subscribe, are $9.99 for 20.

Frequently asked questions

Where do I edit robots.txt in Webflow?

In Site settings, under SEO. Changes go live when you publish the site.

Can Webflow block AI bots without writing rules?

Webflow’s help center describes traffic control toggles in Site settings that generate robots.txt rules for groups of known bots. Check which bots a toggle covers in your own dashboard before relying on it.

Does blocking a crawler in robots.txt remove my pages from AI answers?

It stops a crawler that obeys robots.txt from fetching new copies. It does not remove content already collected, and it does not affect crawlers that ignore the file.

Sources

Platform documentation was read on 2026-10-08. Platforms change these pages and settings without notice, so check the original before you act on a detail.

• Webflow Help Center — Set robots.txt rules
• Webflow — Working with Cloudflare to give website owners more control over AI bots
• RFC 9309 — Robots Exclusion Protocol
• Crawlwise — The AI Crawler Access Report