If you ran an audit on the top 500 SEO-aware sites in 2026, the majority of them now have an llms.txt file at the root. We checked a sample of 80 across our client portfolio and competitor lists last week. 61 of them ship one. None of them, when we asked, could point to a single confirmed citation lift that came from the file itself.
That is the strange status of llms.txt right now. Search demand is steady at 5,400 queries per month on Google in the US according to DataForSEO, peaking around 8,100 last summer. "what is llms.txt" gets 480 to 720 monthly searches. "llms.txt generator" pulls 590. The topic is not going away. And yet the actual evidence that the file does anything is almost entirely vibes.
We have been shipping llms.txt files across client sites since late 2025. Six months of data, a lot of crawler log analysis, and a few honest conclusions. Here is the practitioner take.
What llms.txt Actually Is
For anyone catching up, llms.txt was proposed by Jeremy Howard at Answer.AI in September 2024. The idea is simple. Drop a markdown file at the root of your site, structured a bit like a table of contents, that points AI crawlers at the canonical, summarised version of your most important content. Optionally, a longer llms-full.txt carries the actual content in clean markdown so the model does not have to wrestle with your JavaScript-rendered DOM.
Think of it as a sitemap for language models. Not for indexing in the classic crawler sense. For inclusion in training data or retrieval-augmented generation systems that synthesise answers from web content.
The format is plain markdown. An H1 with the site name, a blockquote with a short description, a few sections with linked references. That is it. There is no XML, no schema, no validation step. You can write one in fifteen minutes.
Who Actually Reads It
This is where it gets messy.
Google has publicly said they do not use it. John Mueller went on the record in September 2025 saying Google does not currently process llms.txt for AI Overviews, AI Mode, or anything else. That position has not changed at the time of writing.
Anthropic ships one for Claude. They host one at anthropic.com/llms.txt and reference it as part of their public documentation. Whether ClaudeBot actually prioritises it during crawls of other sites is not documented anywhere we trust.
OpenAI has not confirmed it. ChatGPT search crawls the open web through OAI-SearchBot. We have not seen any public statement from OpenAI that the file changes crawl behaviour or citation weight.
Cloudflare added native support. They will autogenerate one from your site structure if you flip a setting in their AI Crawl Control product. That is a strong signal that infrastructure providers are betting on the standard, even if the model providers are not yet committing.
Mintlify, Vercel, and a long tail of doc platforms generate one by default. This is where adoption is highest. Developer documentation sites have a clean structure that maps well to the format, and they have the most to gain from being the source the model picks.
Mark this for what it is. The biggest player in AI search has said no. The next biggest has said nothing. The standard exists because a slice of the community, including some infrastructure platforms, decided to act before the model providers confirmed they would honour it.
What We Have Seen in Six Months of Testing
We rolled llms.txt out across 14 client sites between November 2025 and February 2026. We watched server logs, AI bot user-agent hits, and AI Overview and AI Mode citation share for the same period. Three patterns came out of it.
1. Crawl Hits on the File Are Real, But Modest
GPTBot, ClaudeBot, and PerplexityBot have all hit llms.txt on our client sites. Server logs show consistent fetches, usually once a week, sometimes more on high-traffic domains. Google-Extended hits it too, but that does not mean Google is using the content for anything user-facing. They are storing it. What happens after is opaque.
For context, the same bots hit robots.txt an order of magnitude more often. So whatever weight llms.txt is being given, it is not being treated as a primary signal.
2. No Detectable Citation Lift, Standalone
We A/B tested in the closest way you can with crawler-driven systems. Seven of the 14 sites got the file. Seven did not. Same vertical pairings where possible, same on-page work otherwise. After 16 weeks, the citation share difference in AI Overviews, AI Mode, ChatGPT, and Perplexity for the group running llms.txt was inside noise. Not zero, but not significant.
That does not mean the file does nothing. It means the file alone is not moving the needle in any clean, measurable way at the level of granularity we can track.
3. Where It Might Help: Disambiguation and Structure for Smaller Models
The one place we saw a faint signal was on long-tail product and documentation queries on smaller models that use RAG against the open web, including some of the newer enterprise search tools. The llms.txt file gave the retrieval system a cleaner shortcut to the right canonical URL. We are talking about handfuls of mentions, not a flood. But the pattern was consistent enough to note.
If you are running a documentation-heavy site, a product catalog, or anything where a user is asking an AI assistant to look up a specific named thing on your site, the file may be quietly pulling weight. We would call this plausible, not proven.
Why It Has Become a Default Anyway
Given that the evidence is so soft, why is adoption so high?
A few reasons stand out, none of them stupid.
The cost is near zero. A small markdown file at the root of a domain takes less effort to ship than a single internal link. There is no real downside, except that maintaining a stale one is worse than not having one.
It serves as a public signal. Shipping llms.txt is a statement that you take AI search seriously. Clients ask for it. Audit tools flag its absence. Whether it does anything technical or not, it has become a marker of being a serious operator in 2026 SEO.
It is forward-looking insurance. The model providers might start honouring the file at any point. If Google flips the switch tomorrow, the sites that already ship a clean one will be a step ahead. That is a cheap option to hold.
It is a reasonable place to put guidance the bots might use anyway. Even if no model explicitly prioritises llms.txt, the structured summary of your site, with clear titles and the entity names you want to be associated with, ends up in the crawl. That content has to land somewhere in the training mix or retrieval index. We suspect, with low confidence, that it nudges entity association at the margin.
Our Take
We are still recommending llms.txt to clients in 2026, with caveats. The cost is so low that not shipping one is harder to justify than shipping a reasonable version. But anyone selling you llms.txt implementation as a citation-driving tactic is overselling.
The work that actually moves citations is the same as it was when we wrote our GEO playbook for Google AI Mode: clean passage-level answers, entity clarity across the site, original information, sensible schema, and crawl access for the AI bots that matter. The file is a footnote on that list.
If you have an afternoon to spend on AI search optimisation, do not spend it perfecting your llms.txt. Spend it rewriting your top three commercial pages to lead with cite-worthy 50-word answer passages. The return is an order of magnitude better.
How We Are Implementing It
Since we still recommend it, here is the version we are actually shipping. Nothing fancy.
# SEO Turtle
> Data-driven SEO agency focused on AI search optimisation, technical SEO,
> and generative engine optimisation for B2B, legal, financial, and igaming clients.
## Core Services
- [AI Search Optimisation](https://seoturtle.com/services/ai-search-optimization): GEO for Google AI Mode, ChatGPT, Perplexity, and Claude.
- [Technical SEO](https://seoturtle.com/services/technical-seo): crawlability, Core Web Vitals, schema, JavaScript SEO.
- [Link Building](https://seoturtle.com/services/link-building): editorial outreach, digital PR, niche placements.
## Key Resources
- [Google AI Mode GEO Playbook](https://seoturtle.com/seo-insights/how-to-rank-in-google-ai-mode-geo-playbook)
- [AI Crawlers and robots.txt Guide](https://seoturtle.com/seo-insights/ai-crawlers-robots-txt-guide)
- [AI Search Engine Market Share](https://seoturtle.com/seo-insights/ai-search-engine-market-share-chatgpt-google-gemini)
## About
SEO Turtle is an independent SEO agency. Founded in Cyprus, serving clients globally.
A few practical notes on what we have learned.
Keep the link list tight. A bloated llms.txt with 200 entries is worse than a focused one with 15. Lead with the pages you most want associated with your brand identity. Treat each entry like prime homepage real estate.
Sync it with your sitemap priorities. If your sitemap promotes a page and your llms.txt ignores it, you are sending mixed signals. We script ours from the same source as the XML sitemap, with manual curation of the order.
Date it and version it. A dateModified comment at the top is not officially part of the spec, but it costs nothing and lets you track when you last touched it. Stale llms.txt files are a real problem we keep finding on competitor audits.
Decide on llms-full.txt deliberately. The long form file with full content is a bigger commitment. For documentation-heavy sites, ship it. For marketing sites, probably skip it. Maintaining a 200KB full-content mirror that goes out of sync with your live pages is a worse signal than not having one at all.
The Bottom Line
llms.txt is the most-discussed file in SEO right now with the weakest evidence base behind it. Search volume for the term is steady, adoption is high, and the practical impact on citations is, in our six-month testing, indistinguishable from noise on its own.
Ship one anyway. Spend twenty minutes on it, not twenty hours. Then go back to the work that actually moves AI citations, which is the page itself. The standard might harden into something real over the next year. If it does, you will be glad you have a clean file ready. If it does not, you have lost the equivalent of one cup of coffee in implementation time.
The agencies and SEOs who treat llms.txt as either a magic bullet or a waste of time are both wrong. It is a cheap option on a possible future. Hold the option. Do not pay for it twice.
If you want a second pair of eyes on your AI search setup, including whether your current llms.txt is helping or hurting, drop us a line. We will give you a straight answer.
