An AI crawler can only quote what it can read in your page's HTML, and most of them cannot run the JavaScript that many modern websites use to show their content.
If your homepage looks full when you open it in a browser but the text is loaded by scripts, ChatGPT's and Claude's crawlers may see a nearly empty page. That is a build decision, made months before anyone asks whether you appear in AI answers.
This article explains which crawlers visit, what OpenAI says its three bots do, what research found about JavaScript, and a ten-minute test you can run today. It belongs to our wider guide to UK web design agencies, since it is mostly a question of how the site is built.
Which AI crawlers visit a website?
Several do, and they have different jobs: some gather training data, some build a search index and some fetch a page when a person asks about it.
OpenAI documents three. OAI-SearchBot is the one that surfaces websites in ChatGPT's search features. GPTBot crawls content that may be used to train its models. ChatGPT-User visits a page when a person's request in ChatGPT, or a custom GPT, needs it. Other companies run their own, including Anthropic's ClaudeBot and PerplexityBot, and Google's AI features draw on Googlebot's index.
The three OpenAI bots are controlled separately through robots.txt. That matters, because the choice to block training is not the same as the choice to disappear from ChatGPT search.
Should you block or allow them?
For most businesses that want customers to find them through AI tools, allow the search crawler, and decide separately about training.
According to OpenAI, blocking OAI-SearchBot keeps your site out of ChatGPT's search answers, though it may still appear as a plain link. Disallowing GPTBot signals that your content should not be used for training. OpenAI says a change to robots.txt can take around 24 hours to take effect for search.
ChatGPT-User visits are triggered by people, so OpenAI notes that robots.txt rules may not apply to them in the same way.
Make the decision deliberately. Some publishers block training crawlers to protect their content. A local service business gains little from doing so and loses visibility if it blocks the wrong bot. Check your robots.txt for a blanket rule that a security plugin or host added without telling you.
Can AI crawlers read JavaScript?
Mostly no. Research by Vercel and MERJ found that none of the major AI crawlers it tested render JavaScript, which means content that is loaded in the browser is invisible to them.
The tested list included OpenAI's crawlers, Anthropic's ClaudeBot, Meta's crawler, Bytespider and PerplexityBot. The study found that ChatGPT's and Claude's crawlers do download JavaScript files, at 11.5% and 23.84% of requests, but do not execute them. Googlebot is the exception: Gemini uses Googlebot's infrastructure, which renders JavaScript fully.
Vercel's advice to site owners is to server-render the content that matters: the main text, the metadata and the navigation. That can be done by rendering pages on the server, generating them at build time or refreshing them on a schedule. Client-side rendering is fine for extras such as a chat widget.
Many small business sites get this right without trying, because WordPress and similar systems send full HTML. The risk is highest with sites built as single-page apps, with page builders that load sections with scripts, or with booking and review widgets that supply your most persuasive content.
We explain how the framework choice affects this in why we build on Next.js, and when we do not.
How reliably do AI crawlers fetch pages?
Not very. The same study found that more than a third of requests from ChatGPT's and Claude's crawlers hit pages that did not exist.
xychart-beta
title "Share of crawler requests returning a 404 error (%), Vercel and MERJ study"
x-axis ["Googlebot", "Claude", "ChatGPT"]
y-axis "Percent of requests" 0 --> 40
bar [8.22, 34.16, 34.82]
Vercel reported 404 rates of 34.82% for ChatGPT and 34.16% for Claude, against 8.22% for Googlebot. The study does not say why, but stale or guessed addresses are an obvious candidate.
The practical lesson is to keep your URLs stable, keep redirects short and direct, and keep your sitemap accurate. A site that has changed its page addresses three times in five years, with redirect chains from each version, is harder for an AI tool to read than one that has kept its structure.
Our article on why redesigns lose traffic for three months covers the redirect side.
How do you test what an AI crawler sees?
View the page without JavaScript and check that your main text, headings and contact details are still there.
Run this ten-minute check on your homepage and your top two service pages:
- Open the page in a browser, right-click and choose "View page source" (not "Inspect"). Search for a distinctive sentence from the page. If it is there, the text is in the HTML.
- If you cannot find it, switch JavaScript off in your browser's developer settings and reload. Anything that disappears is invisible to crawlers that skip scripts.
- Check robots.txt at yourdomain.co.uk/robots.txt. Look for lines that disallow OAI-SearchBot, GPTBot, ClaudeBot or PerplexityBot, or a blanket disallow.
- Open your sitemap and click ten addresses. Each should load without redirecting more than once.
- Search your server logs or your host's analytics for the bot names above, if you have access. It shows who is visiting.
If step one fails for your main content, raise it with your developer. It is a build issue, and fixing it is usually cheaper than anything done at the content level.
What does a sensible robots.txt look like?
For a business that wants AI search visibility but does not want its content used for training, the file is short.
The file has three blocks:
User-agent: OAI-SearchBotfollowed byAllow: /User-agent: GPTBotfollowed byDisallow: /User-agent: *followed byAllow: /and aSitemap:line pointing at your sitemap address
This allows ChatGPT's search crawler, asks OpenAI not to use the site for training and leaves other crawlers alone. Each AI company documents its own user agent names, so check them before you copy anything, and remember that robots.txt is a request that well-behaved crawlers honour, not a lock.
Whether to block training is a business choice, and there is no right answer for every firm.
Could your security settings be blocking the bots?
Yes. Firewalls, bot protection and some security plugins block AI crawlers by default, so a site that welcomes them in robots.txt can still turn them away at the door.
Vercel's write-up notes that its platform offers a firewall rule to block AI bots, which is the kind of switch that gets turned on once and forgotten. Content delivery networks, hosts and WordPress security plugins offer similar controls.
If your robots.txt is open but your server logs show no AI bot visits at all, ask your host or developer whether a rule is blocking them.
What about text inside images and PDFs?
A crawler reads text, not pictures, so anything that matters should also exist as real text on the page.
A banner that says "Free consultation, 20 years' experience" as part of an image tells a crawler nothing. The same applies to a price list saved as a picture, or an opening-hours graphic.
PDFs can be read by many tools, but a PDF is a poor home for your main service information. Put it on a page, and offer the PDF as a download.
Alt text on images, which also helps screen reader users, gives a crawler a short description. We list it among the five accessibility fixes most UK SME sites need.
What else helps an AI tool use your page?
Clear structure helps: one topic per page, descriptive headings, short paragraphs that answer the heading, and facts stated plainly.
Being readable is a precondition. Being quotable is the next step, and it depends on how each passage is written. Our guide to passage-level extractability explains how to write sections that stand alone, and the piece on llms.txt for a small business covers whether that file is worth your time.
Structured data, accurate business details and consistent names across the web matter as well. A professional services firm that wants to be named for a specialism should state that specialism in text on a crawlable page, not only in an image or a script-built banner.
Is this worth doing for a small site?
Yes, because it costs little at build time and gives you a page every crawler, search engine and screen reader can read.
The same fixes help accessibility and speed. A page that sends clear HTML first is easier for assistive technology and faster on a weak phone signal, which we cover in page speed as a sales metric.
Do not expect miracles from crawlability alone. Whether an AI tool names your business also depends on what other sites say about you, a point we examine in what 45 GEO studies actually found. Readability gets you into the running.
Ask your developer
Send the five-step test to whoever built your site and ask for a written answer. If your content is built in the browser, our website design service can rebuild it so the text arrives in the HTML, and you can contact us to talk it through.
