Someone in your business has typed "best accountants in Northampton" into ChatGPT, seen that you were not in the answer, and treated that as a finding.
Or they saw that you were, and treated that as a win. Both readings are wrong for the same reason. A generative engine does not give the same answer twice, so one check is a coin toss dressed up as data, and AI visibility tracking has to be built to cope with that.
This piece explains why the outputs move, what a measurement has to vary across before it means anything, and a protocol small enough for a UK firm to run without buying software. It also shows how to read a citation rate once you have one.
Why the same prompt gives different answers
The same prompt returns a different set of sources from one run to the next because the engine samples its words, re-runs retrieval each time, and may or may not search the web at all. A 2026 survey of generative engine optimisation research by Olivier Martinez describes the whole thing as "a stochastic, partially observable pipeline" and reports that commercial audits show "substantial run-to-run variability". We summarise what all 45 generative engine optimisation studies found separately; this piece takes the measurement problem on its own.
The numbers behind that phrase are large. One study summarised in the survey's measurement section found day-to-day source overlap of roughly 0.34 to 0.42 (a Jaccard score, where 1.0 means identical), with similar figures for repeat runs inside 24 hours. Another found that repeated runs at temperature zero, the setting that is supposed to make a model deterministic, still changed 9 to 28% of decisions.
Part of the reason is that the model does not always look anything up. The same section reports that 57.8% of ChatGPT repetitions in one study did not activate web search. When the search step fires, the answer leans on live pages. When it does not, the answer comes from training data, and the cited sources change accordingly.
Then there is drift over weeks. SISTRIX tracked 82,619 prompts across 17 weeks in six countries including the UK, on Google AI Overviews, Google AI Mode and ChatGPT Search. ChatGPT Search replaced 74% of the domains it cited from one week to the next. AI Mode replaced 56%. AI Overviews were the stable one, and even there the cited set was completely unchanged for only just over half of queries.
"For projects, this means basing success metrics on presence over time, not on individual results." Johannes Beus, SISTRIX, AI citation drift, May 2026
So the person who checked once and did not see you may have hit a run where search did not fire. The person who checked once and did see you may have hit one of the 26% of citations that survived the week. Neither of them knows which.
What a measurement has to vary across
A usable measurement has to vary along four axes: run, paraphrase, date and engine. That is the minimum the survey sets out ("a GEO measurement must therefore vary along at least four dimensions: run, paraphrase, date, and engine"), and each axis fixes a specific blind spot.
Repeated runs deal with the sampling noise above. Kevin Indig's prompt tracking method uses five runs per prompt per platform each week, and he is blunt about the alternative: "Reporting a point estimate from one run is astrology." The Schulte study in the survey proposes seven to eight repetitions as a starting point, while noting that no number is a universal standard.
Paraphrases deal with the fact that buyers do not share one wording. The survey recommends three to five paraphrases per information need. The arithmetic matters here: ten phrasings of the same question are one intent sampled ten times, not ten intents, so you report them together.
Dates deal with drift. If 74% of ChatGPT's citations turn over weekly, a monthly check misses most of what happened. Limy, one of the tracking tools, re-runs its prompts every 24 hours for this reason. Weekly is a reasonable floor for a manual protocol.
Engines deal with the fact that they do not agree. The survey notes that "a domain cited by Perplexity may never appear in ChatGPT", and points to a 2024 study of Bing Chat and Perplexity where only 26% of the 355 domains appeared in both engines. Indig gives a worked case: one brand mentioned in 78% of problem-stage prompts on ChatGPT and 34% on Perplexity. Pool those into one score and you have described neither engine.
The fifth thing, which is not an axis but a comparison, is a control. The survey's recommended design is "an untreated baseline, an intervention, and, where possible, a placebo". In plain terms: measure some prompts or pages you do not touch, the same way and at the same time, so that when everything rises together you know the engine moved and you did not.
A protocol you can run without software
Run ten buyer prompts, in three paraphrases each, five times each, on four engines, once a week. That is 600 answers per cycle, 150 per engine, and it takes an afternoon with a spreadsheet and a timer.
- Write ten prompts a buyer would actually type. Weight them towards problem and comparison questions, the kind that decide where law firms and similar firms win or lose a client ("which conveyancing solicitor in Milton Keynes handles leasehold flats"). Keep brand-name prompts to one or two; buyers who already know your name are not the ones you are measuring for. Indig's panel is 40 prompts with 16 of them problem-stage; ten is the small-firm version.
- Write two more phrasings of each, so you have 30 prompt strings. Keep them as a buyer would write them, including the clumsy ones.
- Pick four engines and keep them separate: ChatGPT, Perplexity, Google AI Overviews (or AI Mode) and Gemini is the common set. Use a logged-out or clean session so personalisation does not pollute the sample.
- Run each of the 30 strings five times per engine. Space the runs by a few minutes at least. Do not reword a prompt mid-cycle.
- Choose a control. The simplest is five prompts for a service you offer but are not going to optimise for during the test, run identically. A second option is to track two or three named competitors in the same answers, which gives you a denominator for share.
- Freeze the whole set for the measurement window. Cloro's sample size guide puts it well: adding prompts mid-window "manufactures a trend". Date every change.
Record four things per answer. Whether your business was mentioned by name (yes or no). Whether your site was cited as a source, as a link or footnote (yes or no, with the URL). Your position among the businesses named, counting from one. And the sentiment of the mention, which you can keep to three bins: recommended, neutral, warned against. Keep the raw answer text too. You will want to re-read the ones where you appeared in a "firms to avoid" list.
This matches how the commercial tools work internally, and you can build the whole thing in a spreadsheet before paying for one. Keyword.com's support documentation describes its detection rate as the percentage of runs where the brand appears, with average position and a top-three rate alongside, and it warns that "slight variation between runs is normal". You are doing the same job by hand.
How to read a citation rate
Citation rate is the number of answers that linked to your site divided by the total answers for that engine, and it is only meaningful with its margin of error attached. Mention rate uses the same denominator with a looser numerator: named at all, linked or not.
Here is what one cycle might look like. These figures are illustrative, made up to show the shape of the output. They are not from a Growthcart client or from any real test.
| Engine (150 answers each) | Mentioned | Mention rate | Cited with link | Citation rate | Avg position when mentioned | Warned against |
|---|---|---|---|---|---|---|
| ChatGPT | 42 | 28% | 18 | 12% | 2.6 | 1 |
| Perplexity | 63 | 42% | 51 | 34% | 1.9 | 0 |
| Google AI Overviews | 30 | 20% | 27 | 18% | 2.1 | 0 |
| Gemini | 24 | 16% | 9 | 6% | 3.4 | 2 |
| All four pooled | 159 | 26.5% | 105 | 17.5% | 2.3 | 3 |
Illustrative figures only, labelled as such. The pattern they show is the point: the pooled row describes none of the engines. Perplexity cites almost everyone it mentions, ChatGPT names you without linking two times in three, and Gemini barely knows you exist.
Now the margin. Cloro's guide gives two anchors from the standard interval formula: 72 answers resolve a rate to roughly plus or minus 10 percentage points, and about 300 answers to plus or minus 5. At 150 answers per engine you sit between the two, so a 28% mention rate on ChatGPT is really "somewhere in the low twenties to mid thirties". Detecting a move from 20% to 30% with confidence needs about 294 answers in each period.
That sets the decision rule. A change inside the margin is noise. A change outside it, seen on two consecutive weekly cycles, is a signal. A change outside it on one engine and not the others is a signal about that engine, not about your content.
The control is what stops you taking credit for the weather. Suppose after four weeks your tracked prompts move from 28% to 36% on ChatGPT. If the control prompts moved from 20% to 29% over the same weeks, most of your lift is the engine citing more local businesses in general. The survey makes exactly this point about a published case where "a large raw increase may arise from platform growth".
If budget forces a choice between more prompts and more runs, Cloro's data favours prompts, because the variance between questions is bigger than the variance within one. Ten prompts at five runs beats five prompts at ten.
What this means for buying the work
Any agency or tool reporting AI visibility as a single number from a single run is selling you a screenshot. Ask what the denominator is, how many runs went into it, whether engines are separated, and what the control was. If the answer to the last one is a blank look, the before-and-after chart is unreadable.
The same test applies to us. The AI visibility work we do reports mention and citation rates per engine with the run counts shown, against a frozen prompt set and a control, and we would rather show a flat line with honest error bars than a rising one without them. Our generative engine optimisation service starts from that baseline, because you cannot improve a rate you have not measured.
A single check tells you one thing: what one run of one engine said on one afternoon. Everything after that is counting, and the counting is the job.
If you want a baseline built this way for your firm, accountancy practices included, or want us to check how an existing report was put together, get in touch about AI visibility tracking.