Does llms.txt affect Google Search? What Google says, and what it is for

llms.txt became one of the most-recommended “AI SEO” fixes of 2025, usually with no source behind the recommendation. Google has since answered the question directly for its own products, so this one can be settled with documentation rather than opinion.

What is llms.txt?

A Markdown file, usually at /llms.txt, proposed by Jeremy Howard in September 2024. The proposal’s stated aim is to “provide information to help agents use a website”: a title, a short summary, and lists of links to the pages an agent should read, often pointing at Markdown versions of documentation. It is a proposal published at llmstxt.org, not an IETF or W3C standard.

# Example Co
> Example Co makes invoicing software for small agencies.

## Docs
- [Getting started](https://example.com/docs/start.md): install and first invoice
- [API reference](https://example.com/docs/api.md)

What does Google say about llms.txt?

Google’s AI optimization guide says: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search,” and that such files “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Its AI features page adds that there are “no additional requirements to appear in AI Overviews or AI Mode.” That covers Google Search, AI Overviews and AI Mode. It is a statement about Google, not about every AI product.

A file nobody documents reading is documentation, not optimization. Publish it if it helps people and agents; do not expect it to move a ranking.

Adnan Arodiya, Crawlwise

Does any other AI engine use llms.txt?

We could not find documentation from OpenAI, Anthropic, Perplexity or Microsoft saying their search crawlers read llms.txt or use it to rank or cite pages. Their crawler pages talk about robots.txt and user-agent tokens instead. Some coding agents and documentation tools read llms.txt when a user points them at it; that is a product feature of those tools, not a search signal. If that changes, the vendor’s own docs are where it will show up first.

Should you publish one anyway?

It is cheap and harmless, and for a developer-documentation site it can be genuinely useful to people who paste your docs into an agent. For a marketing site, the time is better spent elsewhere. Crawlwise’s own audit stopped requesting llms.txt by default for exactly this reason: a check that cannot be tied to a documented outcome should not cost you points. Our llms.txt generator is still there, marked experimental, for teams that have a specific integration in mind.

What should you fix instead?

• Crawler access. In the Crawlwise AI Crawler Access Report (top-10000 run, 2026-Q4: the first 10,000 domains of the Cisco Umbrella top-one-million list, each probed once on October 1, 2026), 4,159 of 10,000 domains (41.6%) refused at least one AI crawler. The list ranks hostnames by DNS popularity, so it includes API, CDN and update hosts that never serve a web page, and only the homepage path was probed. Treat the shares as a description of that list, not of the whole web. A refused crawler cannot read any file, llms.txt included. Start with checking whether AI crawlers can reach you.
• Content in the server HTML. Text that only appears after JavaScript runs may be invisible to crawlers that do not render.
• Snippet controls. In Google, nosnippet and max-snippet limit what AI Overviews can use — details here.
• Accurate structured data where it applies — see structured data in the AI search era.

How Crawlwise tests this

The audit does not score llms.txt, because no major search engine documents using it. What it does test is whether the crawlers that feed AI answers can fetch your page at all: the AI crawler checker runs that on one URL for free, and the robots.txt tester shows which rule decides access for a given crawler.

Every verdict, what counts as a block, and what the checks cannot see are written up on the methodology page.

What a plan actually costs

The free tools stay free. You can open them, run them, and leave without an account. A plan is for the moment you want the report saved, a few sites watched, or the same checks from the API. The price on this table is the price at checkout. A yearly plan is ten months of the monthly price, so two months are on us.

A single-page audit is about 1.25 credits, and that includes the live probe of answer-engine crawlers. You see the estimate before anything runs. If a hold is not used, it comes back to your balance. Extra credits, when you already subscribe, are $9.99 for 20.

Frequently asked questions

Will an llms.txt file hurt my site?

Not in Google Search: Google says these files neither harm nor help visibility or rankings because Search ignores them. The cost is keeping the file accurate.

Does ChatGPT or Claude read llms.txt?

Neither OpenAI’s nor Anthropic’s crawler documentation mentions llms.txt. An agent may read it if a user or tool points it there, but we cannot cite a documented ranking or citation effect.

Is llms.txt a replacement for robots.txt?

No. llms.txt describes content; it does not grant or deny access. Crawler access is still set by robots.txt and your CDN.

Sources

Vendor documentation was read on 2026-10-05. Vendors change these pages without notice, so check the original before you act on a detail.

• Google Search Central — AI optimization guide
• Google Search Central — AI features and your website
• llmstxt.org — The /llms.txt file (proposal by Jeremy Howard)
• OpenAI — Overview of OpenAI crawlers
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity — Perplexity crawlers
• Crawlwise — The AI Crawler Access Report (top-10000 run, 2026-Q4)