robots.txt is a plain text file at the root of your website telling automated visitors which parts they may request. It is the single highest-consequence file on most websites, because one wrong line can remove you from search results entirely.
Why one line matters so much
The file is tiny and the syntax is unforgiving. A stray Disallow can take a whole site out of search, and it is a genuinely common accident: a staging site is built with everything blocked, it goes live, and nobody remembers to change the file.
Nothing warns you. The site works perfectly for humans. Traffic simply stops arriving, and by the time anyone investigates, weeks have gone.
What it does not do
It is not security. It is a request, honoured by well-behaved crawlers and ignored by everyone else. Listing a private area in robots.txt tells anyone reading the file exactly where to look, which is the opposite of the intended effect.
It also does not reliably remove a page from search results. A blocked page can still appear if other sites link to it, because the crawler is told not to fetch it, not that it does not exist. Keeping something out of results needs a noindex instruction on the page, which means the crawler has to be allowed to fetch it and read that instruction.
What should be in it
For most businesses, very little. Allow everything, and point at your sitemap so crawlers can find your pages. That is a complete and correct robots.txt for the majority of websites.
Add exclusions only for genuine reasons: internal search results, endless filter combinations, admin paths that would waste crawl effort. If you cannot articulate why something is blocked, it probably should not be.
Read it before you trust it
One subtlety worth knowing: a robots.txt that times out or returns a server error is not the same as a site with no robots.txt. A 404 genuinely means nothing is restricted. An unreadable file means unknown, and careful crawlers treat that conservatively.
Where this shows up in the tracker
Every term in this glossary is something the product actually measures or acts on. Plans start at £29 a month, with rank tracking, AI visibility, the daily AI readiness scan, security and domain health, Search Console, backlinks and audits on every one of them.
Related terms
Training bot vs search bot
AI companies run several kinds of crawler. Search and user-fetch crawlers are how you get into AI answers, so blocking them removes you from those answers. Training crawlers collect material to train models, and blocking those is an editorial choice with no effect on your visibility.
Agent readiness
Agent readiness is a score for how well AI systems can find, fetch and make sense of your website. It covers discoverability, crawler access, machine-readable content and agent capabilities, and it is the first thing to check when you are missing from AI answers.
llms.txt
llms.txt is a proposed standard: a plain-text file at the root of your website that gives AI systems a clean, structured summary of what the site contains and where the important pages are. It is a convention, not a rule, and support for it is not universal.
Canonical
A canonical tag is a line in a page telling search engines which address is the official version, for when the same content is reachable at several different URLs.