When someone wants a plumber, a wedding venue or a fireworks display these days, there is a decent chance they do not search Google at all. They ask ChatGPT, or Gemini, or Perplexity, and they get back a paragraph with two or three business names in it. If yours is one of them, wonderful. If it is not, you never even knew the conversation happened.

Here is the uncomfortable part: many businesses are absent from those answers not because the AI judged them and found them wanting, but because the AI literally cannot read their website.

The one-line mistake

Every website has a small file called robots.txt that tells visiting software what it may read. A few years ago, lots of site owners (and some agencies, and some plugins) added blanket rules blocking AI crawlers. At the time that felt prudent. But the crawlers have since split into two very different jobs, and one blanket rule now blocks both:

  • Training bots (like GPTBot) collect pages to train future models. Blocking these is a legitimate editorial choice, and it does not hide you from anyone.
  • Search bots (like OAI-SearchBot and PerplexityBot) fetch pages to answer live customer questions. Blocking these removes you from AI answers entirely, the same way blocking Googlebot would remove you from Google.

We have seen well-run local businesses with excellent reviews sitting behind a robots.txt that blocks every AI bot ever named. Their competitors, often objectively weaker businesses, get recommended instead, simply because the AI could read their sites.

Three checks you can do today

You do not need a consultant for this. Visit yourwebsite.co.uk/robots.txt and look for lines mentioning GPTBot, OAI-SearchBot, PerplexityBot or Claude. Ask ChatGPT a question a customer would actually ask, and see who gets named. And check whether your key facts (what you do, where, for how much) exist as real text on your pages rather than only inside images or embedded widgets, because crawlers cannot read those.

Or let software do it: our agent-readiness scan runs every one of those checks daily, for free, and scores your site 0 to 100. When a plugin update quietly rewrites your robots.txt in November, you find out the next morning, not the next quarter.