Does my small business need llms.txt? A straight answer

No. Your small business does not need an llms.txt file, and adding one will not get you into Google, ChatGPT or Perplexity.

Google has said in writing that its Search ignores the file. No other AI company has confirmed using it for anything. The server-log studies published this year show the main AI crawlers almost never ask for it.

That said, the file costs about fifteen minutes and does no harm. If a plugin or your developer wants to add one, let them. Just do not pay for it, and do not let anyone tell you it is the reason your business is or is not showing up in AI answers.

The rest of this article covers where the idea came from, what the evidence says, how to add one if you still want to, and the three things that make a far bigger difference to whether an AI tool can read your site at all.

What llms.txt is

An llms.txt file is a plain Markdown text file placed at yourdomain.co.uk/llms.txt that lists the pages on your site you would most like an AI tool to read. It was proposed by Jeremy Howard of Answer.AI on 3 September 2024, and the proposal describes it as "a standardised way to provide information to help agents use a website".

The format is simple. One H1 heading with your business name. A short summary in a blockquote. Then H2 sections, each holding a bulleted list of links with a one-line note on what each page contains. An "Optional" section at the end holds the pages an agent can skip.

Howard's reasoning was that web pages are built for people, wrapped in navigation, adverts and JavaScript, and that an AI with a limited context window would do better with a short, curated map. The proposal was revised to version 2 in August 2026 and notes that its heaviest use is software documentation, where coding agents follow it to find API references.

That last point matters for a small business. The file was designed with developer docs in mind. A plumber in Kettering or a solicitor in Northampton is not the use case, and the evidence below reflects that.

What Google and the AI companies have said

Google has said, in three separate places, that it does not use llms.txt. The first was John Mueller on Bluesky in June 2025, reported by Search Engine Roundtable:

"FWIW no AI system currently uses llms.txt. It's super-obvious if you look at your server logs."

John Mueller, Google Search Advocate, Bluesky, 17 June 2025

A few weeks later Gary Illyes told a Google Search Central event in the Asia Pacific region that Google "doesn't support LLMs.txt and isn't planning to", according to Kenichi Suzuki's account published by Search Engine Land.

Then on 15 June 2026 Google put it in the documentation. Its guide to generative AI features now states: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It adds that maintaining an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search". The changelog entry for that date is headed "Clarifying guidance on llms.txt files".

OpenAI, Anthropic and Perplexity have published nothing that says their crawlers read it. OpenAI's crawler documentation describes three user agents and how robots.txt controls them. It does not mention llms.txt. Anthropic's crawler page is the same. Silence is not denial, but after two years it is telling.

What the server logs show

Four published log studies in 2026 all point the same way: AI crawlers fetch ordinary pages constantly and fetch llms.txt almost never.

The Ahrefs study is the most detailed. Of the requests that did reach an llms.txt file, 96% came from bots, and 77% of those bots were not AI tools at all. The biggest group was SEO audit software checking whether the file existed. GPTBot, OpenAI's training crawler, accounted for about 4.5% of requests. OAI-SearchBot, the one that feeds ChatGPT search, accounted for 0.74%. ClaudeBot was at 0.8%.

The detail that settles it for me is the 404 data. Ahrefs looked at requests for llms.txt files that did not exist. Those requests came almost entirely from humans. Not one came from an AI bot. If OpenAI or Anthropic were looking for the file, their crawlers would try the URL on sites that lack it, the same way Googlebot requests robots.txt on every domain. They do not.

The Ahrefs authors also noted that their customers skew technical, so the 28% adoption figure is a ceiling. Across UK small business sites the real figure is far lower, which is another reason no AI company has built anything that depends on it.

Who the file is for

The people who get something out of llms.txt are documentation sites and developer tools whose users are coding agents. Ahrefs found Claude Code, Anthropic's coding tool, was the second most active named AI fetcher after GPTBot, and agent traffic was the largest AI category at 10.5% of requests.

That makes sense. A developer asks a coding agent how to call a payment API, the agent finds the provider's llms.txt, follows the link to the right Markdown page and gets a clean answer. Stripe-style documentation benefits. A page listing your opening hours and three services does not, because no agent is navigating it that way.

There is also a small downside Ahrefs raised: a hand-written file goes stale, and an llms.txt that points to deleted pages or old prices is worse than none. They also flag a prompt-injection risk: some files try to give the model instructions instead of describing the site.

The 15-minute version if you want one anyway

If you still want the file, here is the shortest honest version. It takes about fifteen minutes and should be done once, then left alone or regenerated by your CMS.

1. Open a plain text editor and type a single line starting with # followed by your business name, for example # Hartley Plumbing, Northampton.
2. Underneath, add a blockquote starting with > that says in one or two sentences what you do, where and for whom.
3. Add an H2 line, ## Services, then a bulleted list of your main service pages as Markdown links, each followed by a colon and a one-line description. Five to ten links is plenty.
4. Add a second H2, ## About and contact, with links to your about page, contact page and any pricing page.
5. Add ## Optional with links to blog posts or case studies an agent could skip.
6. Save the file as llms.txt and upload it to the root of your site so it loads at yourdomain.co.uk/llms.txt. On WordPress, Yoast SEO can generate one and refreshes it weekly, which is better than a static file because it does not go stale.
7. Check it loads in a browser as plain text, then check it again in six months.

Do not pay anyone to do this. If an agency quotes you for an "AI readiness file" or lists llms.txt as a deliverable in an AI visibility package, ask them which engine has confirmed reading it. They will not have an answer.

What matters far more than llms.txt

Three things decide whether an AI tool can read your site, and none of them is a text file. They are your robots.txt rules, whether your content is served as HTML, and whether your pages are indexed by Google.

Allow the bots you want

AI tools cannot cite a page their crawler was blocked from. The most common problem we see on small business sites is a security plugin, a hosting firewall or a well-meaning developer blocking every bot that is not Googlebot. That quietly removes the business from ChatGPT search and Claude.

The crawlers are documented and controlled through robots.txt like any other. OpenAI runs three, and its documentation says each setting is independent: you can block GPTBot from training on your content while still letting OAI-SearchBot show you in ChatGPT search. Anthropic runs three as well.

A sensible position for most UK small businesses is to allow the search and user agents and make a separate decision on the training crawlers. Check your live robots.txt at yourdomain.co.uk/robots.txt, and check the firewall rules in your hosting control panel too, because Cloudflare and similar services can block bots before robots.txt is ever read.

Serve the content as HTML, not JavaScript

None of the major AI crawlers run JavaScript. Vercel's analysis of its own network traffic in December 2024, covering 569 million GPTBot requests and 370 million from Claude in a single month, found that GPTBot, ClaudeBot and PerplexityBot fetch JavaScript files but do not execute them. In their words, the crawlers "can't read client-side rendered content".

For a small business this is the hidden killer. If your site is built on a page builder or a JavaScript framework that loads the text after the page arrives, an AI crawler sees an empty shell. Googlebot renders JavaScript and copes. The AI crawlers do not. Your developer should be able to tell you in one sentence whether the site is server-rendered. If they cannot, view the page source in your browser and search for a sentence from your homepage. A missing sentence means the crawler sees a blank page, and that is the first thing to fix.

The same Vercel study found that ChatGPT's crawler hit a 404 page on 34.82% of its fetches, against 8.22% for Googlebot. AI crawlers are worse at handling broken links and redirects than Google is. Clean internal links and a current sitemap do more for AI visibility than any dedicated AI file. Our technical SEO work starts with exactly these checks.

Be indexed and eligible for a snippet

For Google's AI Overviews and AI Mode, the only technical requirement is the one that applies to ordinary Search. Google's guide says a page "must be indexed and eligible to be shown in Google Search with a snippet". It also notes that the site must be included in Search generative AI features in Search Console, a control that someone may have switched off.

There is no separate AI index. If a page does not appear for a site: search, it cannot appear in an AI Overview, and no file at the root of the domain changes that.

The decision rule

Spend your time where the crawlers spend theirs. Every log study this year shows AI bots requesting ordinary HTML pages by the hundreds of millions and llms.txt by the dozen. If your robots.txt allows the search bots, your text is in the HTML, and your pages are indexed, you have done the part that counts. That order, retrieval before polish, is the same one the survey of 45 generative engine optimisation studies arrives at. Add the file afterwards if you like, in the quarter of an hour it takes, and then forget about it.

If you want someone to check robots.txt, rendering and indexing for your site and tell you plainly which AI tools can currently read it, our LLM optimisation team does that as a fixed-scope review. Get in touch.