AI search, evidenced
llms.txt, and whether it does anything
llms.txt is a proposed markdown file that describes your website to a language model. The question is whether anything reads it. Across 137,210 domains, 28% publish an llms.txt file and 97% of those files received no requests at all in a month, and Google states that Search does not use files like it.
28%
of 137,210 domains publish an llms.txt file.
97%
of those files received zero requests in a month.
0
requests from AI bots for llms.txt on sites that don't have one. They are not looking.
Source: Ahrefs server-log analysis of 137,210 domains that received traffic in May 2026. Of the requests that did reach an llms.txt file, 96% came from bots and 19.5% of fetches came from named AI tools.
What llms.txt actually is
llms.txt is a markdown file at the root of a website, proposed by Jeremy Howard on 3 September 2024, to give a language model a compact description of the site at inference time. The specification requires an H1 with the project name, which is the only required section, a blockquote summarising it, and optional H2-delimited sections listing links. The idea is sound: a model reading a tidy index instead of a navigation-heavy homepage.
# Babel42
> Babel42 measures whether AI assistants recommend your brand to buyers.
## Docs
- [How AI Visibility works](https://babel42.io/ai-visibility): what the product measures
- [Metrics explained](https://babel42.io/ai-visibility/metrics): every reported number, definedNote what it is not. llms.txt does not control access, does not block anything, and does not replace a sitemap. Those are three different files doing three unrelated jobs, and conflating them is the most common mistake on this topic.
llms.txt vs robots.txt vs sitemap.xml
| File | What it does | Is it read? | Does it matter? |
|---|---|---|---|
| robots.txt | States which crawlers may fetch which paths. | Yes. Every major AI crawler documents that it follows it. | Decisively. Blocking the wrong crawler removes you from an assistant's answers. |
| sitemap.xml | Lists the URLs you want discovered. | Yes, and it is how crawlers find pages that are not prominently linked. | Yes, and dead entries are expensive: ChatGPT's crawler already spends 34.8% of fetches on URLs that no longer exist. |
| llms.txt | Describes your site to a language model in markdown, at inference time. | Almost never. 97% of published files received zero requests in a month. | Not measurably, today. |
What Google says about llms.txt
“You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.”
Google's Gary Illyes has said publicly that Google does not support llms.txt and has no plans to. John Mueller compared it to the keywords meta tag, which search engines abandoned because it was trivially manipulated, and noted that you can tell from server logs that AI services do not even check for the file.
So should you add one?
Add one if it takes ten minutes and you would enjoy having it. It is cheap, it is harmless, and if adoption ever turns into consumption you are already there. What it should not do is come off your list ahead of anything measured. The single most deflating finding in the data is not that the files go unread; it is that AI bots never requested an llms.txt from sites that did not publish one. They are not looking for it and finding it missing. They are not looking.
What to do instead, in order
Check which crawlers you actually block.
OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. That is a documented consequence, unlike anything llms.txt offers. Blocking GPTBot, which only collects training data, is a different and much smaller decision.
Make sure your pages carry content without JavaScript.
No major AI crawler executes JavaScript. If your pages are assembled in the browser, assistants receive an empty shell no matter how good your llms.txt is.
Clear dead URLs out of your sitemap.
ChatGPT's crawler spends 34.8% of its fetches on URLs that no longer exist, against Googlebot's 8.2%. Every stale entry costs you a page that could have been read.
Write sections that stand up alone.
Retrieval matches a question against one section at a time. A section that opens on this or however, or that owns no text under its heading, cannot be quoted usefully however it was discovered.
Babel42's free AI SEO audit checks all four, and reports whether you publish an llms.txt without ever counting its absence against your score.
llms.txt, answered
What is llms.txt?
llms.txt is a markdown file at the root of a website that describes the site to a language model at inference time. Jeremy Howard proposed it on 3 September 2024. The format requires an H1 with the site name, a blockquote summarising it, and optional H2 sections listing links. It is unrelated to robots.txt, which controls crawler access.
Does llms.txt actually work?
Not on the published evidence. Across 137,210 domains, 28% publish an llms.txt file and 97% of those files received no requests at all during a full month. Of the requests that did land, 96% came from bots and under a fifth from named AI tools. No major AI company has committed to reading the file.
Does Google use llms.txt?
No. Google's AI optimisation guidance states that you do not need to create machine-readable files, AI text files, markup or markdown to appear in Google Search including its generative AI features, because Search does not use them. Gary Illyes said Google does not support llms.txt and has no plans to, and John Mueller compared it to the long-abandoned keywords meta tag.
What is the difference between llms.txt and robots.txt?
robots.txt controls access and llms.txt offers a description. robots.txt tells a crawler which paths it may fetch, is honoured by every major AI crawler, and blocking the wrong agent in it can remove you from an assistant's answers entirely. llms.txt is a markdown summary intended to help a model use your site, and almost nothing currently requests it.
Should I add an llms.txt file?
Adding one is cheap and harmless, so there is no strong argument against it if it takes ten minutes. There is also, on current data, no measured benefit, and Babel42's free audit reports whether you have one without ever counting its absence against your score. If the choice is llms.txt or fixing pages that arrive empty without JavaScript, fix the pages.
Will llms.txt matter later?
Possibly, and that is a genuine reason to keep an eye on it rather than a reason to prioritise it now. The finding that most undercuts it is behavioural rather than technical: AI bots never requested an llms.txt from sites that did not have one, meaning they are not looking for the file and finding it missing. They are not looking.
See what assistants actually get from your site.
A free audit that fetches your pages the way an AI crawler does, reports what came back, and tells you which findings are measured and which are guesswork.