llms.txt is a proposed standard: a plain-text file at the root of your website that gives AI systems a clean, structured summary of what the site contains and where the important pages are. It is a convention, not a rule, and support for it is not universal.
What it is for
A modern web page is mostly not content. It is navigation, cookie banners, scripts, related-article widgets and footers, wrapped around the part that actually says something. A system trying to understand your site has to strip all that away and guess at the structure underneath.
llms.txt is an attempt to hand over that structure directly: here is what this site is, here are the important pages, here is a sentence about each. Markdown, no markup, no navigation, nothing to strip.
What it is not
It is not robots.txt and it does not control access. Nothing in llms.txt grants or denies permission to anything; crawler access is decided in robots.txt, and that is where a mistake actually costs you.
It is not a ranking factor, and nobody should tell you it is. It is also not a substitute for the pages themselves: a beautiful llms.txt pointing at pages that do not answer the question achieves nothing.
And it is not universally consumed. Adoption is real but partial. Anyone claiming a guaranteed effect on how assistants treat your site is overselling it.
So is it worth adding?
Yes, with realistic expectations, on two grounds. It costs very little to produce, particularly if you generate it from the same page list your sitemap uses so it cannot go stale. And the exercise of writing one sentence per page describing what that page is for is genuinely clarifying: if you cannot write the sentence, the page probably has a problem.
Treat it as cheap housekeeping that might help, not as a lever.
What goes in one
A heading with the site name. A short summary of what the business does, in prose. Then sections of links, each with a sentence explaining what that page covers. Some sites add a section of notes: facts they want stated accurately, or limits worth knowing.
Ours is generated from our route list, so adding a page to the site adds it to the file automatically. That is the part worth copying, whatever you think of the standard.
Where this shows up in the tracker
Every term in this glossary is something the product actually measures or acts on. Plans start at £29 a month, with rank tracking, AI visibility, the daily AI readiness scan, security and domain health, Search Console, backlinks and audits on every one of them.
Related terms
robots.txt
robots.txt is a plain text file at the root of your website telling automated visitors which parts they may request. It is the single highest-consequence file on most websites, because one wrong line can remove you from search results entirely.
Agent readiness
Agent readiness is a score for how well AI systems can find, fetch and make sense of your website. It covers discoverability, crawler access, machine-readable content and agent capabilities, and it is the first thing to check when you are missing from AI answers.
AEO
AEO stands for answer engine optimisation. It is the work of getting your business named and cited when an AI system answers a question, rather than merely ranking on a page of links. The name is new; a good deal of the work is not.
Training bot vs search bot
AI companies run several kinds of crawler. Search and user-fetch crawlers are how you get into AI answers, so blocking them removes you from those answers. Training crawlers collect material to train models, and blocking those is an editorial choice with no effect on your visibility.