llms.txt Adoption: An Honest Reality Check

Here is the honest state of llms.txt in August 2026: the spec is real, the tooling ecosystem is real, fetches of the file show up in real server logs - and not one major AI engine has publicly committed to consuming it for answers or weighting it in retrieval. Both halves are true at once, and most coverage picks a side instead of holding both. Advocates talk as if llms.txt were a ranking lever; skeptics talk as if an unconsumed file were a scam. The accurate position is narrower: llms.txt is a near-zero-cost bet with a verifiable floor and an unverified ceiling.

We recommend shipping one, and we say that while running a business that benefits when you believe llms.txt matters - so this post separates what you can verify yourself from what nobody can currently verify, and is explicit about which claims are which.

What llms.txt is designed to do

The llms.txt specification, proposed by Jeremy Howard of Answer.AI in September 2024, addresses a genuine mechanical problem: LLMs operate under context-window limits, and parsing a modern site - navigation, cookie banners, JS-rendered templates - burns tokens and loses structure. The spec asks you to publish a single markdown file at /llms.txt containing a short description of the site and a curated, annotated link list pointing at your most important content, with an optional /llms-full.txt carrying expanded flattened content for direct ingestion.

Think of it as a curated table of contents for machine readers - closer in spirit to a sitemap than to robots.txt, despite the name. It grants no permissions and blocks nothing; robots.txt remains the access-control layer. If the concept is new to you, our plain-English llms.txt explainer covers the format in detail.

The confirmed column

Three things about llms.txt are verifiable today, two of them without leaving your own server.

The spec and its ecosystem exist. The proposal is public and stable, and a real tooling layer grew around it: documentation platforms like Mintlify generate llms.txt automatically for hosted docs, generators and validators are plentiful, and directory sites track thousands of published files. Notably, AI vendors themselves publish the file for their own docs - Anthropic maintains one at docs.anthropic.com/llms.txt - which tells you the companies building the models consider the format at least useful enough to serve. Publishing for your own docs is not the same as committing your crawler to consume everyone else's, and it would be sloppy to conflate the two. But it is adoption on the supply side, and it is checkable: fetch the URL.

Some crawlers fetch it. This is the claim you can verify in your own access logs. Sites that publish llms.txt observe requests for the file from various AI-associated user agents and from the fetch traffic of assistant sessions. Grep your logs for GET /llms.txt and you will see whether anyone is asking for yours. In our audits we commonly see fetches within weeks of publication. A fetch is not proof of consumption - a crawler can request a file and discard it - but "nobody ever asks for it" is now empirically false for many sites, and your logs settle the question for yours.

It costs almost nothing. One static markdown file, no build complexity, no render cost, no interaction with your robots.txt or your rankings. The entire downside case is the hour it takes to write and the ongoing minute it takes to keep current.

The unconfirmed column

Now the other ledger. No major answer engine - not OpenAI, not Google, not Anthropic, not Perplexity - has publicly documented that its production retrieval or answering pipeline consumes llms.txt, prefers sites that publish it, or weights it in citation selection. Google representatives have publicly expressed skepticism that AI systems use the file at all. The absence of commitment is not proof of absence - vendors rarely document retrieval internals, and pipelines change without announcement - but the claim "llms.txt improves your AI visibility" is currently unproven, and anyone selling it as a proven lever is ahead of the evidence.

You will also see the Duda finding quoted as that proof, so let's frame it precisely: Duda's 2026 analysis of local-SMB sites found that sites with llms.txt correlated with more AI-referral sessions. That is a correlation on one platform's sites, the published study does not state its measurement window, and the causal arrow is unresolved - sites diligent enough to ship llms.txt are diligent about schema, content structure, and everything else that plausibly drives AI referrals. It is an interesting signal consistent with the optimistic story. It is not evidence that adding the file causes traffic, and we will not cite it as if it were. The full context sits with the rest of our sourced numbers on the AI search statistics page.

Why we still say ship it

Given an unproven upside, the recommendation hinges entirely on the cost side, and the cost side is where llms.txt is unusual. The decision math looks like this.

The floor is zero. No engine penalizes the file's presence, it cannot conflict with robots.txt because it grants nothing, and no crawler behaves worse because you published a markdown index. A bet with no downside needs only a plausible upside, not a proven one.

The upside is plausible on mechanism. Context-efficient site summaries solve a real problem for machine readers, vendors already publish the format themselves, and agent-style tools that fetch pages on demand - as opposed to big-index engines - benefit most from exactly this kind of curated map. If any category of consumer quietly adopts it, early publishers win retroactively.

The bet is log-verifiable. Unlike most AEO advice, this one reports back. Publish the file, then watch your access logs: you will know within weeks whether crawlers fetch it, which agents, how often. You are not asked to take adoption on faith; you are asked to spend an hour and then read your own evidence.

Curation has a second use. Writing llms.txt forces you to decide which twenty pages define your site - an exercise that routinely exposes gaps and stale flagship content. Even in the world where no engine ever reads the file, you read it.

When you do ship it, do it properly: the step-by-step generation guide covers structure and common mistakes, and if you go further, the llms-full.txt guide covers the expanded variant. Then validate the result - format, reachability, content-type - with our free llms.txt checker.

How to hold the claim honestly

If you write about llms.txt for clients or stakeholders, calibrate the language. Say "we published llms.txt; fetches are visible in our logs" - verifiable. Say "no engine has committed to consuming it; we treat it as a cheap option on future adoption" - accurate. Do not say "llms.txt improves AI rankings" - unproven - and do not present the Duda correlation as causation. This discipline is not pedantry; it is the difference between advice that survives a skeptical CTO and advice that torches your credibility the first time someone checks. The overclaiming around llms.txt has probably done more to slow its serious evaluation than the skeptics ever did.

Frequently asked questions

Does any AI engine officially use llms.txt right now?

No major engine has publicly committed to consuming llms.txt in its answering or retrieval pipeline, and Google representatives have voiced skepticism that AI systems use it. What is confirmed is the supply side: the spec, wide tooling support, vendors publishing the file for their own docs, and fetch requests visible in site logs.

Will llms.txt hurt my site if engines ignore it?

No. It is a passive markdown file that grants no permissions and blocks nothing, so the worst case is that it goes unread. Access control stays in robots.txt, and rankings are unaffected by the file's presence. That zero-downside profile is the core of the case for shipping it.

What did the Duda study actually show?

Duda's 2026 analysis of local-SMB sites found that sites with llms.txt correlated with more AI-referral sessions. It is correlational, single-platform, and the published study does not state its measurement window - so it cannot establish that adding the file causes traffic. Treat it as a directional signal, not proof.

How do I check whether my llms.txt is being fetched?

Grep your server access logs for requests to /llms.txt and /llms-full.txt and note the user agents. Fetches typically appear within weeks on sites that get regular AI crawler traffic. To confirm the file itself is valid and reachable, run it through a checker before you start watching logs.

Verify yours in thirty seconds

The reality check cuts both ways: llms.txt is neither the ranking hack the hype claimed nor the pointless gesture the backlash claimed. It is a cheap, honest bet that reports its own results. If you have one, point our free llms.txt checker at it to confirm it parses and resolves; if you don't, the guides above will have one live today. And if you want the file in context - alongside your schema, crawlability, and answer extractability - a full Citevera audit scores the whole stack and tells you which fixes actually move your citation readiness, with the honest caveats attached. That honesty is the product.