How many questions does an AI visibility measurement need?

This is a property of how the assistants work rather than a fault in any one of them. A model samples from a range of possible replies, so two runs of the same prompt take two different paths through it. Our repeat study counted how often that changes the names an assistant gives. Where an assistant reads the live web as it answers, the pages it read this morning are not the pages it read last week, and the answer moves with them.

The consequence for measuring AI visibility is the whole of this page. A single answer is one draw from a distribution. It is evidence that the answer is possible. It is not evidence of what a buyer is usually told.

The band adds two spreads, each taken as a standard error: how far the assistants differ from each other, and how far your questions differ from each other. The sum is scaled to a ninety five percent interval. A category where every assistant says roughly the same thing about every question produces a narrow band. One where an assistant puts you first and another leaves you out, or where you win some questions and vanish from others, produces a wide one, and that width is a finding rather than a defect in the run.

Three properties of it are worth knowing. It is floored at one point, so the product can never report absolute certainty about a figure derived from answers that vary. Where fewer than two assistants answered there is no spread to compute at all, so no band is published and the run states that rather than printing a zero. And a change between two runs on the same questions is judged against the spread between the assistants alone, because the questions are the same in both.

Three things have to hold before a set of answers is worth calling a reading.

• Repeats. Every question goes to each assistant several times on a full run, so one unusual answer cannot set the score by itself.
• Breadth. The same question set goes to ChatGPT, Google Overview, Gemini, Copilot, Claude, Perplexity and Grok. A brand can be the obvious recommendation in one of them and missing from another.
• The disagreement is kept. It goes into the band around the Index instead of being smoothed into an average, because how far the assistants differ is the part a buyer actually experiences.

An assistant that did not answer is disclosed rather than filled in. The Index is computed from the assistants that did answer, the shortfall is stated beside it, and an answer from an earlier run is never reused as a fresh one.

A search that comes back with no AI answer, because Google showed plain results or Copilot gave no answer, is not a shortfall. That search is left out of the assistant's figures rather than counted as an answer that did not name the brand, and a question none of whose searches came back with one is marked No AI answer.

That is the shape of the arithmetic and it is as far as it can honestly be taken. What number of questions gives you a band you can act on depends on things that differ by brand: how far the assistants disagree in your category, how many of your questions carry buying intent rather than research intent, and how small a monthly movement you need to be able to call. The second matters because a recommendation rate counts only the buying questions, so a set weighted toward research leaves fewer answers behind it.

Three limits here are fixed. A question set holds at least ten buying questions, because across the runs Delphi has measured, a set of five let the recommendation rate land about twelve points either side of where a different five would have put it, and ten narrowed that to about eight. It is capped at 40 questions and a run at 840 answers, because every question is put to every assistant several times, and the size of the set is what a run costs. Inside those, the set is yours to edit, every run is pinned to a version of it, and the trend view marks a comparison between runs that crosses a change to it.

The figure worth asking any tool for is not the question count. It is the number of answers behind the score, and the band that number produced.

The test is deliberately blunt. If the size of the change is larger than the spread between the assistants it is reported as a confirmed change. If it is not, the dashboard and the history page tell the reader that the figures moved by less than the margin of error of the measurement itself, and that it is too early to call any of it a result.

There is a third answer and it is not no. Where the band could not be computed, because too few assistants answered, the dashboard says it could not tell whether the figure moved. Could not tell and did not move are different statements, and collapsing them lets a product report a result it never measured.

Two further cases are refused rather than reported. Where the way answers are read or scored changed between two runs, the step between them is not published as a movement, because it was made by the method rather than by the brand. And three of the five steps carry no margin of their own: Seen and Chosen each have an observable that can be split by assistant, while Found, Trusted and Bought do not, so a change on those three is never called a confirmed result, and the run says it could not tell rather than reporting a verdict it did not measure.

These are properties of the method rather than gaps waiting to be filled, and they are the reason the rest of the page is worth anything.

• A sample, not a census. The reading covers the questions in your set on the day the run happened. A question nobody put in the set was not measured.
• Personalisation and memory. An answer shaped by the history of one person is not reproducible, and no attempt is made to model it. What is measured is the answer to a fresh question with no earlier conversation.
• The assistants change underneath the measurement. A model update can move a score with nothing changed on your side. Every answer records the model that produced it wherever the assistant names one, so a movement caused by a model change can be told apart from your own progress.
• Retailer owned assistants are not read. Nothing here measures what an assistant inside a retailer tells a shopper, and no figure implies otherwise.
• No commercial outcome is claimed. Nothing here says a position caused revenue or what a piece of work earned. Where you give a revenue band, Delphi estimates how much of it is decided in answers you do not win, as a range with its basis, and that estimate never enters the Index.

Take them to any tool, this one included. Each has an answer a vendor can give in a sentence, and an evasion that is easy to recognise once you are listening for it.

• How many answers is this computed from? Questions, assistants and repeats, as three numbers rather than one.
• What is the range around it? A figure with no range has not been checked against how much the answers behind it vary.
• What happens if you ask the same question again? If the answer is that nothing changes, either the sampling is very deep or it happened once.
• What do you report when a movement is smaller than the range? The honest answer is no confirmed change, and it is the one nobody enjoys giving.

Our own answers are on this page and in the method, in the same words. A confidence band is published beside the Index, the dashboard and the history page report a movement smaller than that band as no confirmed change, and where the band could not be computed the dashboard says so instead of quietly reporting stability.