How does AI read your website, and what can it see?
It reads what your server sends: the HTML, the headings, the text, and the markup that names your business. Google says browser agents also inspect the page structure, the accessibility tree and sometimes a screenshot 1. Anything drawn by JavaScript after the page loads, or a phone number that lives only in an image or a chat widget, may never reach it.
Mr Goh's back hurt and he wanted Saturday morning
Mr Goh's lower back had been sore since Monday, and he wanted it seen before the weekend was gone. He asked ChatGPT:
Find a physiotherapist near Raffles Place with a slot on Saturday morning and book it for me.
The AI goes straight to the clinics' own pages, takes what each server sends, and drops any page that gives it nothing it can use.
A physiotherapist near Raffles Place had a free slot that morning. If its site never said so in text an AI could read, Mr Goh was booked elsewhere before anyone there knew he had asked. Is your practice in that position? Are the clinics down the road already filling Saturdays?
Not long ago Mr Goh would have typed "physiotherapist near me" and phoned the first result. Now an agent can take the whole task for him 1. So what does an agent get when it opens your page? We measured it on 21 small-business sites, and the answer is narrower than what a person sees.
What 21 sites in Hong Kong and Singapore showed
We did not test whether agents visited; the table measures what was there to read.
Eight sites served a page with no outline
The first finding is about JavaScript, the code that runs in the browser after the page arrives. On 8 sites that code draws the headings and body text. The server sends a near-empty shell (the raw HTML, the first thing any fetcher receives), and the words appear only once a script runs. A person never notices. A fetcher that does not run scripts receives a page with no outline, like a book whose chapters are added after you leave.
Five more were borderline, with part of the outline missing (R2). Ask your web person whether the headings are readable in the page source. If not, the fix is server rendering or pre-rendering.
Ten sites never said what the business is
The second finding is about markup, the hidden labels that tell a machine what something is. On 10 sites we found no Organization or LocalBusiness block, nothing an agent could resolve to a name and an address (EN1).
Two smaller gaps: seven sites publish content nobody signed (EE1), and seven have no contact route as text, no address, phone or email an agent could copy (EE4).
Three kinds of visit, and one robots.txt
Your robots.txt is a plain file that tells crawlers (automated visitors) what they may fetch. Each visitor announces itself with a user-agent name; the vendors publish them, in three kinds 2345.
| Visit | User-agents | What it does | If you block it |
|---|---|---|---|
| Retrieval crawler | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot | Builds the index an AI assistant cites from | You leave the answers |
| User-triggered fetcher | ChatGPT-User, Claude-User, Perplexity-User | Fires when a person asks about your page | Perplexity says its fetcher generally ignores robots.txt 5 |
| Training crawler | GPTBot, ClaudeBot, Google-Extended | Model training and grounding | Costs you nothing in citations |
Most "block the AI bots" advice was written for the third row and applied to all three. OpenAI's own instruction for appearing in ChatGPT search is one line: allow OAI-SearchBot to crawl the site 6.
What Google says an agent is
Google's guide on its generative AI features defines the term: "AI agents are autonomous systems that can perform tasks on behalf of people, such as booking a reservation or comparing product specifications." On how they see your site: "Browser agents may access your website to gather the data they need to complete these tasks, such as analyzing visual renderings (like screenshots), inspecting the DOM structure, and interpreting the accessibility tree" 1.
The DOM is your HTML as the browser holds it; the accessibility tree is the page as a screen reader sees it. An agent gets nothing you meant but did not put on the page.
What to do, in order
- Keep the retrieval crawlers allowed. To opt out of training, block GPTBot, ClaudeBot and Google-Extended by name only (AI2, AI4).
- Serve headings and body text in the HTML. If JavaScript draws them, turn on server rendering or pre-rendering. The free scan shows whether this is your case (R1, R2).
- Declare the business once. An Organization or LocalBusiness block with legal name, address and your identity links (EN1, EN2).
- Use the same name everywhere. Title, footer, schema and directory listings, with English and Chinese names in the same Organization block (EN3).
- Sign it and make yourself reachable. An author on what you publish; phone, email and address as text beside any WhatsApp button, which an agent cannot use (EE1, EE4).
What we did to our own site
We built the scanner so an AI assistant can use it without a person in the loop. The For AI assistants page describes one request that scans any public site and returns both scores, three findings and our price as data.
On 13 September 2026 a DeepSeek chat session, asked by its user about a website, called that endpoint on its own and presented both scores with a link back to us. It found nothing it had to guess, which is the whole argument of this article.
Ranking still works, a little less every year
The old route still works for a site that ranks. Fewer each year: for United States Google searches, 68% ended with no click at all in early 2026, where 49% did in 2019 7. More patients and customers take an answer, or send an agent, and it quotes whichever business wrote its facts into the page.
Moving from a site that ranks to a site an agent can read is our job. Start with the free scan of your own site: it lists what an agent finds when it opens your pages.
Questions people ask
Should I block GPTBot?
A decision about training, not visibility. GPTBot is OpenAI's training crawler; OAI-SearchBot is the one that puts you in ChatGPT search results 26. Block the first if you wish and keep the second allowed. A wildcard block takes out both.
If I block ClaudeBot, will Claude stop citing me?
No. Anthropic documents ClaudeBot for training and Claude-SearchBot for search, controlled separately in robots.txt 3. Blocking the training crawler leaves the search one alone, so your citations stay as they are.
Can an agent read my PDF menu or price list?
Sometimes, and slowly. Text in a PDF is readable if it was typed rather than scanned; prices in an image are not. Put the menu or price list on a page as text, with the PDF beside it.
Do I need an llms.txt for this?
Not for the visits described here. None of the four vendors' crawler documentation mentions the file 2345, and Google says Search does not use it 1. Our llms.txt article covers it.
Can an agent book with me if I only have WhatsApp?
Not today. Google's example of an agent task is booking a reservation 1, and the agents that do it work through booking systems and web forms, not chat apps. If bookings matter, a form or booking widget an agent can operate is worth more than the button.
How can I see whether agents visit my site?
In your server logs, by user-agent: the names in the table above. The free scan does not read your logs; it tells you what an agent will find when it arrives.
Next: Do you need an llms.txt file?
Sources
- Google Search Central, Optimizing your website for generative AI features on Google Search, 10 July 2026. Accessed 13 September 2026.
- OpenAI, Overview of OpenAI crawlers, undated. Accessed 13 September 2026.
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, 7 April 2026. Accessed 13 September 2026.
- Google Search Central, Overview of Google crawlers and fetchers, 14 July 2026. Accessed 13 September 2026.
- Perplexity, Perplexity crawlers, undated. Accessed 13 September 2026.
- OpenAI Help Center, Searching the web with ChatGPT, undated. Accessed 13 September 2026.
- SparkToro, In 2026, less than one third of Google searches still send a click, 9 June 2026. Accessed 13 September 2026.
Our own counts come from our batch of 9 September 2026, 21 sites, one visit each, scored with the published rubric. Not a market survey.