AI companies run several kinds of crawler. Search and user-fetch crawlers are how you get into AI answers, so blocking them removes you from those answers. Training crawlers collect material to train models, and blocking those is an editorial choice with no effect on your visibility.
The three jobs
Search crawlers build the index an assistant consults when answering questions. If one cannot reach your pages, your pages are not available to be cited.
User-fetch crawlers fetch a page at the moment a person asks about it, because someone pasted your link or asked about your company directly. Blocking these means an assistant cannot read your site even when a customer explicitly asks it to.
Training crawlers gather text used to train future models. This is the one people mean when they talk about AI scraping their work, and it is a genuine editorial and commercial decision.
Why the distinction gets missed
They come from the same companies and the names look similar, so a well-meaning attempt to keep a business out of AI training frequently blocks the search crawler by mistake. The result is exactly backwards: the content still contributes nothing to training you objected to, and the business quietly disappears from the answers customers see.
It also happens the other way, through copy-pasted robots.txt files from a blog post, where somebody blocks everything AI-shaped without knowing what each entry does.
How to decide
If you want customers to find you through AI assistants, allow the search and user-fetch crawlers. There is no good argument for blocking those unless you actively want to be absent.
Training crawlers are a real choice and it depends on your business. A publisher whose product is the writing itself has an obvious reason to say no. A local service business generally has nothing to protect and little to gain from blocking.
What matters is deciding on purpose, rather than discovering in six months that one line in one file has been quietly costing you.
Where this shows up in the tracker
Every term in this glossary is something the product actually measures or acts on. Plans start at £29 a month, with rank tracking, AI visibility, the daily AI readiness scan, security and domain health, Search Console, backlinks and audits on every one of them.
Related terms
robots.txt
robots.txt is a plain text file at the root of your website telling automated visitors which parts they may request. It is the single highest-consequence file on most websites, because one wrong line can remove you from search results entirely.
Agent readiness
Agent readiness is a score for how well AI systems can find, fetch and make sense of your website. It covers discoverability, crawler access, machine-readable content and agent capabilities, and it is the first thing to check when you are missing from AI answers.
llms.txt
llms.txt is a proposed standard: a plain-text file at the root of your website that gives AI systems a clean, structured summary of what the site contains and where the important pages are. It is a convention, not a rule, and support for it is not universal.
AEO
AEO stands for answer engine optimisation. It is the work of getting your business named and cited when an AI system answers a question, rather than merely ranking on a page of links. The name is new; a good deal of the work is not.