Adoption of llms.txt jumped nearly ninefold over the past year, yet server logs show almost none of that effort reaches an AI chatbot. A June 2026 tracking study from Originality.ai, covering more than 3 million websites, found llms.txt usage rose 8.8x between June 2025 and May 2026. A companion server-log analysis cited through that same report found that 97% of llms.txt files received zero AI requests in May 2026.
That gap between adoption and actual use is the real story behind llms.txt in 2026. Site owners keep adding the file because GEO guides and SEO tools recommend it. The crawler data behind it tells a much flatter story. Below, I separate what has been verified from what is still guesswork, myth by myth.
What llms.txt actually is
llms.txt is a plain Markdown file, usually placed at a site’s root, that lists links and short summaries meant to help large language models understand a site’s content. Developer Jeremy Howard proposed the format in 2024 as a lightweight alternative to scraping full HTML pages.
The idea borrowed the simplicity of robots.txt. A site drops a text file in one place, and any AI system that wants a clean, structured version of the site’s content can read it instead of parsing navigation menus and ad scripts. In theory, that saves the crawler work and gives the publisher a say in what gets summarized.
A fuller variant, llms-full.txt, bundles entire page contents into one file instead of just links and summaries. A third format, ai.txt, promoted by the licensing group Spawning, takes a different approach and focuses on permissions, telling AI systems which content they may or may not use for training. All three get lumped together under the loose label of AI visibility files, even though they serve different purposes.
The proposal spread fast through GEO content because it was easy to implement and easy to recommend in a checklist. Writing a short Markdown summary takes an afternoon. Verifying whether any AI system actually reads it takes server-log access most writers never check, which is part of why the gap between adoption and use went unexamined for so long.
Myth 1: llms.txt helps you rank or get cited in AI Overviews
This is the most common claim attached to llms.txt, and it does not hold up. Google has said directly that the file plays no role in Search. According to Search Engine Roundtable’s coverage of Google’s John Mueller, Mueller wrote on Bluesky in 2025 that no AI system currently uses llms.txt, and that none of the major consumer chatbots fetch the file at all.
Mueller’s comment covered Google’s own AI Overviews and AI Mode, which rely on the existing Search index rather than a separate feed from this file. If it had no effect on a page’s presence in that index, it would not change whether the page gets cited. Site owners chasing AI Overviews visibility this way are, by Google’s own account, optimizing for a signal the system does not read.
Myth 2: AI crawlers are reading the files that websites publish
Adoption charts make this file look like a fast-growing standard. The request logs behind those files tell a different story. The Ahrefs server-log data cited in the Originality.ai update, drawn from roughly 137,000 domains, found that identifiable AI retrieval bots accounted for just 1.1% of all requests made to llms.txt files in May 2026.
Training crawlers fared only slightly better. GPTBot made up 4.51% of requests, ClaudeBot 0.80%, and DeepSeek’s bot 0.02%. The largest single group of visitors to these files, at 21.7%, were SEO audit tools checking whether the file existed at all, not AI systems reading it for content. Unidentified bots made up another 14.9%, and general-purpose web crawlers added 13.1% more.
Add those categories together and roughly half of all traffic to these files comes from compliance checkers or indiscriminate crawling, not from a language model retrieving the page to answer a question. Originality.ai founder Jonathan Gillham called the pattern concerning, and confusing for site owners who were told the file would help their AI visibility.
The 97% figure deserves one caveat. Server logs capture requests that reach a web server, but some on-demand AI fetches route through proxies or caching layers that can obscure the original user agent. That limitation does not close the gap entirely. Even a generous adjustment for undercounting would not turn a 1.1% AI-bot share into evidence of broad reliance on the file.
Myth 3: Every major AI platform ignores llms.txt the same way
This is where nuance matters, and where myth and fact blur a little. Google’s position is the clearest and most negative: the file will not positively or negatively affect rankings or visibility in Search, full stop. That statement covers AI Overviews and AI Mode specifically.
OpenAI and Anthropic have not issued anything as direct about their own chat products. Mike King of iPullRank has argued that some retrieval-augmented systems, agent frameworks, or future tooling could plausibly read it even if today’s consumer chatbots do not. That is a reasonable hypothesis, not a confirmed behavior.
Spawning, the group behind the ai.txt permissions format, makes a narrower and more verifiable claim: its own licensing API checks ai.txt directives before allowing a model to train on flagged content. That is a specific, named use case, different from the broad assumption that AI assistants read llms.txt by default. Treat any claim that ChatGPT or Gemini reads it for everyday search answers as unverified until a platform confirms it on the record.
Myth 4: Fast adoption growth means llms.txt is becoming standard practice
The growth numbers are real, but the base they are growing from is tiny, and independent trackers do not even agree on the current rate. A separate adoption tracker from Rankability, scanning the top 10,000 domains by traffic as of September 2026, found the file on just 8.3% of those sites, with a further chunk unreadable or blocked during the scan.
That tracker also broke adoption down by industry. Travel sites led at 16.0% adoption among the top 10,000 domains, followed by financial services at 15.4%. Technology sites, despite being closest to the AI conversation, sat at just 6.4%. Rankability describes these category splits as directional estimates rather than firm figures, given the sample size within each industry.
Fortune 500 adoption lags even further behind the hype. A ProGEO.ai study found only 7.4% of Fortune 500 companies had published one as of March 2026, compared with 92.8% that had a standard robots.txt file. For comparison, our look at robots.txt rules for AI crawlers covers a file that nearly every major site already has, and that AI crawlers demonstrably respect, unlike this one.
Myth 5: Having an llms.txt file can’t hurt, so why not add one
This is true as far as it goes. An llms.txt file is unlikely to damage rankings or AI visibility on its own. But harmless is a low bar when teams are spending real hours writing summaries, maintaining the file across site updates, and reporting it up the chain as a GEO win.
My take, based on how most content teams operate under limited headcount: time spent maintaining a file that gets close to zero AI requests is time not spent on things that measurably move AI citations, like clear headings, direct answers near the top of a page, and content that fact-checkers can verify against a primary source. The opportunity cost is the real risk, not the file itself.
What’s confirmed versus what’s still speculation
Confirmed, with a named source: Google’s AI Overviews and AI Mode do not use llms.txt for ranking or visibility, per Mueller’s public statement on Bluesky. Confirmed, with data: AI retrieval and training bots made up a small single-digit share of requests to these files as of May 2026, per the Ahrefs-sourced log analysis. Confirmed: adoption has grown quickly in percentage terms but remains low in absolute terms across the broader web, with trackers disagreeing on the exact current rate.
Still speculation: whether any AI agent framework, browser-based assistant, or future retrieval system will meaningfully rely on this format going forward. Whether it indirectly helps by acting as a tidy summary a human editor, a secondary SEO tool, or a specialized licensing API might reference, the way Spawning’s ai.txt checks already do for training permissions. Neither broader claim has public, verifiable evidence behind it yet, and this guide will need revisiting as platforms publish more on the subject.
How to think about llms.txt without the hype
If a site already has a clean content structure, descriptive headings, and a working robots.txt policy for AI crawlers, adding a short llms.txt file costs little and closes a loose end some GEO checklists ask for. The mistake is treating it as a visibility lever on par with structured data, clear answers, or earning citations the way our guide on how to rank in AI search lays out.
Teams that want to verify any of this for themselves don’t need a third-party study. Server logs already capture bot activity. Filtering a site’s own access logs for user agents like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, then checking how often any of them hit that path specifically, answers the “does it work for us” question directly, using a site’s own traffic rather than someone else’s sample. Most hosting dashboards and CDNs expose raw access logs or a filtered bot-traffic report that makes this a short exercise, not a data-engineering project.
It also helps to separate this question from the broader topic of AI crawler access generally. Google’s recent moves to surface more inline links inside AI Mode, covered in our piece on AI Overviews preferred sources, show where Google is actually investing attention right now, and it is not in a third-party text file format.
The broader lesson applies past this one file. GEO advice often spreads faster than the evidence behind it, because a simple tactic is easier to write about than a nuanced one. Before adopting the next recommended file, badge, or markup pattern, ask which named platform has confirmed it matters, and whether that confirmation covers the specific AI surface a team actually cares about.
Why this matters and what to do next
GEO budgets are finite, and llms.txt has absorbed attention that the data does not yet justify. The file is not harmful, but it is not the visibility shortcut it is often sold as. Spend the bulk of GEO effort on content clarity, verifiable facts, and the technical basics AI crawlers are confirmed to use, and treat llms.txt as a low-priority, low-risk extra rather than a core workstream.
- Check your own server logs for AI bot requests to the llms.txt path before assuming it is being read.
- Keep robots.txt rules for AI crawlers current, since that file is confirmed to matter.
- Prioritize direct answers, clear structure, and sourced facts over writing llms.txt summaries.
- Revisit this topic periodically. Platform behavior around AI crawlers and agents is still young and can change.
Frequently asked questions
Does llms.txt help with Google AI Overviews or AI Mode?
No. Google’s John Mueller has said no AI system currently uses llms.txt, and that it does not affect ranking or visibility in Search, including AI Overviews and AI Mode.
Do ChatGPT or Claude read llms.txt files?
There is no confirmed evidence that consumer-facing ChatGPT or Claude chat interfaces fetch llms.txt during normal use. Server-log data shows AI retrieval bots made up a small share of requests to these files as of May 2026. Some specialized agent or licensing tools may behave differently, but that is not documented publicly for mainstream search features.
Is it worth creating this file anyway?
It is low-risk and low-cost if the content is already organized. It should not replace work on content clarity, structured data, or the technical signals AI crawlers are confirmed to use.
How is llms.txt different from robots.txt?
robots.txt tells crawlers which parts of a site they can access and is widely respected, including by AI crawlers. llms.txt is a separate, voluntary file meant to summarize content for language models, and current evidence shows it is rarely requested by AI systems.
Leave a response