Which AI crawlers should you allow in robots.txt, and which should you block?

Mr Pereira blocked every AI bot, and nobody told him what it cost

On Friday Mr Pereira pastes a list of AI bots into his robots.txt, because a webinar said to block them all. He runs a maths tuition centre in Woodlands and feels careful.

That evening a parent opens Google AI Mode and asks for what she actually wants.

Is there a maths tuition centre near Woodlands with small classes and a Saturday slot?

His centre fits every word. But his list shut out the crawler that fetches pages for ChatGPT search along with the ones that only train models, so the AI has nothing of his to read and names other centres instead. The Saturday seats stay empty, and nothing tells him why. Is your robots.txt doing what his does? Are other centres already being named for questions like that?

A few years ago that parent typed "tuition centre near me" and phoned down the list. Now she describes the class she wants.

Keep the retrieval crawlers in, decide training on purpose

"AI bots" is not one animal. A training crawler reads your site to improve a model. A retrieval crawler fetches your site because somebody just asked a question it answers. The first is a courtesy call; the second is the customer at the door.

What I tell an owner is to treat them separately. Block the training set and nothing measurable happens to your visibility. Block the retrieval set and the engine that wanted to cite you gets nothing to read, attribute or link. The stock advice meant "do not train on me" and shipped "do not cite me", and because the owner's browser never requests robots.txt, the misfire is invisible from inside the business ¹².

The order of work

1. Read your own robots.txt first. Open it in a browser at /robots.txt. Every token on it belongs to one of three families, described further down. Anything you cannot place, look up on the vendor's own page before you trust it (AI1).
2. Check the retrieval five are allowed. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot. Each one blocked is an engine you have switched yourself off in (AI2).
3. Decide the training question on purpose. Blocking is a legitimate choice that costs no citations; allowing is a legitimate choice that keeps your options open. Either way it should be a decision, not an absence (AI4).
4. Re-check after any platform or plugin update. Hosting tools ship AI-blocking toggles that rarely ask which family (AI1).
5. Then make the rest of the site readable. A robots.txt that says yes to a crawler whose fetch finds a JavaScript shell is a yes to nothing. The agent-visit article takes that apart, and the llms.txt one settles what the file with the misleading name is for.

An example robots.txt you can adapt

This is an example, not a prescription: it opts out of training and leaves the retrieval and user-triggered crawlers allowed. If you would rather not opt out, delete the training lines. The OpenAI, Anthropic and Google names come from their own pages ¹²³.

# Opt out of model training. Retrieval crawlers get no rule, so they stay allowed.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot, PerplexityBot,
# Googlebot and Bingbot, may read the site.
User-agent: *
Disallow:

One caution all the vendor pages share: robots.txt is a request that well-behaved crawlers honour, not removal from the web. Our rubric's research note records two training tokens that honour it unreliably, so our check reports training policy as data, never as a verdict.

Three kinds of visit, and what blocking each one does

Our rubric's AI-crawler module reads the robots.txt each engine publishes its tokens against, and sorts every token into one of three families ¹²³⁴.

Google-Extended "doesn't have a separate HTTP request user agent string"; crawling is done with Google's existing crawlers and the token is "used in a control capacity" ³ — a switch over how your content may be used, not a visitor you can meet in the logs. And Perplexity says of its user-triggered fetcher: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules" ⁴. A person sent it. If that visitor must never arrive, robots.txt is not the enforcement; a firewall rule is.

None of the 159 sites blocks a retrieval crawler

Across the 159 Hong Kong and Singapore small-business sites scanned between 9 and 14 September 2026 the picture was one-sided. 141 allow all five retrieval crawlers our check reads, 9 had no robots.txt at all, and on 9 we could not read one when we visited.

Eight sites — 7 in Singapore, 1 in Hong Kong — block the full training set our check tracks and none of the retrieval crawlers: the split, done deliberately and done right. The rest block nothing: citable, with no say recorded on training. Nobody in the batch had made the paste-in misfire yet; the risk is the advice waiting in their next search.

The old way still works, a little less every year

A site that ranks for the short phrase still gets enquiries. It gets fewer as the AI answers more directly: United States Google searches ended without a click 68% of the time in early 2026, and 49% of the time in 2019 ⁵. A longer block list will not change that, and nobody should be paid to "block AI" wholesale: it removes the retrieval family too.

Moving a site from ranking to being read and cited is our job. The free scan reads your robots.txt the way the crawlers do, and shows which of your tokens you meant.

Questions people ask

Should I block GPTBot?

If you want to opt out of OpenAI's training, yes, and it costs you nothing in citations: the fetcher that surfaces sites in ChatGPT search is OAI-SearchBot, a separate token ¹.

Does blocking ClaudeBot stop Claude citing me?

No. ClaudeBot is the training crawler; the search fetcher is Claude-SearchBot, and the user-triggered fetcher is Claude-User — Anthropic documents them as separate tokens ². Block ClaudeBot alone and the citation crawler still reads you.

A fetcher ignored my robots.txt. Is something broken?

Possibly not. Perplexity documents that Perplexity-User "generally ignores robots.txt rules", because a person asked for that fetch ⁴. If a visitor must never arrive, the enforcement is a firewall rule, not a line in robots.txt.

Does robots.txt keep my content out of training?

It is a request, honoured by well-behaved crawlers, not an eraser: nothing already learned is removed. Google-Extended controls whether your content may ground Gemini answers, and is a token rather than a visitor ³.

I have no robots.txt at all. Is that a problem?

Not for citation: with no rules, nothing is blocked, and none of the sites in our batch that lacked one was invisible to the answer engines. What you lack is a recorded say on training crawlers (AI2).

Is Googlebot now an AI crawler?

Google runs both layers of search: the classic index and its AI features, which read what Googlebot fetches ³. Block Googlebot and you leave both layers at once; keep it allowed.

Is this what llms.txt is for?

No. llms.txt is a Markdown index proposed for coding assistants, and no major engine says its crawler reads one. The crawler question is settled in robots.txt,; the llms.txt article has the detail.

Sources

1. OpenAI, Overview of OpenAI crawlers, undated. Accessed 17 September 2026.
2. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, 7 April 2026. Accessed 17 September 2026.
3. Google Search Central, Overview of Google crawlers and fetchers, 14 July 2026. Accessed 17 September 2026.
4. Perplexity, Perplexity crawlers, undated. Accessed 17 September 2026.
5. SparkToro, In 2026, less than one third of Google searches still send a click, 9 June 2026. Accessed 17 September 2026.

Our own counts come from our four sector batches, 9 to 14 September 2026, 159 sites, one visit each, scored with the published rubric. Not a market survey.