llms.txt is a proposed standard: a plain-text file at the root of your website that gives AI systems a clean, structured summary of what the site contains and where the important pages are. It is a convention, not a rule. As of 2026 no major AI platform has said its assistant reads it, and Google says its search ignores it.
What it is for
A modern web page is mostly not content. It is navigation, cookie banners, scripts, related-article widgets and footers, wrapped around the part that actually says something. A system trying to understand your site has to strip all that away and guess at the structure underneath.
llms.txt is an attempt to hand over that structure directly: here is what this site is, here are the important pages, here is a sentence about each. Markdown, no markup, no navigation, nothing to strip. It was proposed by Jeremy Howard of Answer.AI in September 2024.
Who actually reads it
Almost nobody, on the evidence available. Google's own guidance on generative AI features, updated in July 2026, says you do not need to create new machine-readable files, AI text files or Markdown to appear in Google Search, because Google Search does not use them. It adds that such files neither help nor harm your visibility.
Ahrefs checked the server logs of 137,210 domains for May 2026. 97% of the llms.txt files on those sites received no requests at all. Of the requests that did arrive, 96% were from bots, and most of those were SEO audit tools, profiling tools and ordinary web crawlers. The AI retrieval bots behind ChatGPT and Perplexity accounted for about 1%. The one AI tool that fetched it in any number was a coding agent, not a search assistant.
OpenAI, Anthropic and Google all publish an llms.txt for their own developer documentation. That is publishing, not reading. None of them has said their assistant fetches yours.
What it is not
It is not robots.txt and it does not control access. Nothing in llms.txt grants or denies permission to anything; crawler access is decided in robots.txt, and that is where a mistake actually costs you.
It is not a ranking factor, and nobody should tell you it is. It is also not a substitute for the pages themselves: a beautiful llms.txt pointing at pages that do not answer the question achieves nothing.
Anyone claiming a guaranteed effect on how assistants treat your site is overselling it.
So is it worth adding?
Probably not needed, but it will not hurt to have one if it becomes mainstream. It costs very little to produce, particularly if you generate it from the same page list your sitemap uses so it cannot go stale. And the exercise of writing one sentence per page describing what that page is for is genuinely clarifying: if you cannot write the sentence, the page probably has a problem.
Treat it as cheap housekeeping, not as a lever. Our readiness scan checks for it and notes what it finds, but does not score it either way, because the evidence does not support scoring it.
What goes in one
A heading with the site name. A short summary of what the business does, in prose. Then sections of links, each with a sentence explaining what that page covers. Some sites add a section of notes: facts they want stated accurately, or limits worth knowing.
Ours is generated from our route list, so adding a page to the site adds it to the file automatically. That is the part worth copying, whatever you think of the standard.
Sources
Where this shows up in the tracker
Every term in this glossary is something the product actually measures or acts on. Plans start at £29 a month, with rank tracking, AI visibility, the daily AI readiness scan, security and domain health, Search Console, backlinks and audits on every one of them.
Related terms
robots.txt
robots.txt is a plain text file at the root of your website telling automated visitors which parts they may request. It is the single highest-consequence file on most websites, because one wrong line can remove you from search results entirely.
Agent readiness
Agent readiness is a score for how well AI systems can find, fetch and make sense of your website. It covers discoverability, crawler access, machine-readable content and agent capabilities, and it is the first thing to check when you are missing from AI answers.
AEO
AEO stands for answer engine optimisation. It is the work of getting your business named and cited when an AI system answers a question, rather than merely ranking on a page of links. The name is new; a good deal of the work is not.
Training bot vs search bot
AI companies run several kinds of crawler. Search and user-fetch crawlers are how you get into AI answers, so blocking them removes you from those answers. Training crawlers collect material to train models, and blocking those is an editorial choice with no effect on your visibility.