The same question, two different answers
Open ChatGPT, ask it to recommend software in your category, and write down what it says. Close the tab, open a fresh one, ask the exact same question. There's a real chance you get a different list, a different order, and a different set of brands cited. Nobody did anything wrong. That's how these models work.
Answer engines sample from a probability distribution every time they respond. Temperature settings, retrieval that pulls slightly different sources, model updates, personalization, even the time of day all nudge the output. The answer you see is one draw from a range of possible answers, not the answer.
So when a vendor hands you a screenshot that says "look, you're ranked first," the honest question is: first out of how many tries? If they ran it once, they captured a coin flip and framed it as a scoreboard. The same prompt ten minutes later might not mention you at all.
Why a single capture is closer to a rumor than a metric
A measurement has to be repeatable enough that you'd trust it to make a decision. A single screenshot fails that test on its own terms. It can't tell you whether you show up, for example, 90% of the time or 15% of the time, and those are wildly different realities with wildly different fixes.
This is where single-sample reporting quietly misleads. A good screenshot overstates your position. A bad one makes you panic and rewrite pages that were already working. Both happen because one capture has no error bars. You're reacting to noise and calling it signal.
The fix isn't a fancier screenshot. It's volume. If you want to know how an engine actually treats your brand, you have to ask many times and look at the shape of the answers, not pick the one you liked best.
How to score a prompt instead of capturing it
- Pin the exact prompts you care about, worded the way real buyers ask, not keyword-stuffed queries.
- Run each prompt against every engine you track on a fixed daily cadence, so sampling is consistent instead of ad hoc.
- Collect every response as its own data point. Thirty runs of one prompt is thirty observations, not one screenshot.
- Compute the metric across the full set: mention rate, average position, share of citations, and the run-to-run swing.
- Report the distribution, not the highlight. In the illustrative example above, a 63% mention rate with a ±31 point swing is the truth; "ranked #1" is a cherry.
- Re-run on the same schedule and watch the average move over time, so a real improvement is distinguishable from a lucky day.
What Crescive measures instead
Crescive runs every tracked prompt against every engine daily and computes its metrics across the resulting distribution. A score reflects a true average over many answers, not whichever response happened to come back when someone hit refresh.
That changes what the number means. When Crescive says your mention rate went from 41% to 68%, that's across hundreds of samples, and it comes with the variance so you know how solid the gain is. When it drafts a fix and you approve it, the before-and-after is measured the same way on both sides, so the lift is real and not an artifact of two different coin flips.
The point isn't that non-determinism is a problem to eliminate. You can't. It's baked into the engines. The point is that any honest AEO metric has to be built on top of that variability instead of pretending it doesn't exist. A distribution respects how these systems actually behave. A single screenshot bets you won't ask twice.
Key takeaways
- AI answers are non-deterministic, so the same prompt can return different brands, orders, and citations on each run.
- A single screenshot has no error bars. It can't tell you whether you appear, for example, 90% of the time or 15%, which makes it an anecdote, not a metric.
- Credible AEO scoring runs every prompt against every engine repeatedly and reports the average plus the swing, so wins are provable and not luck.
FAQ
Why do AI assistants give different answers to the same question?
Answer engines generate responses by sampling from a probability distribution rather than returning one fixed result. Factors like temperature settings, which sources get retrieved, model updates, and personalization all shift the output, so the same prompt can produce different brand mentions, rankings, and citations from one run to the next.
How should AEO visibility be measured given that variability?
Run each tracked prompt against every engine on a consistent daily schedule, treat every response as a separate data point, and compute metrics across the full set, including mention rate, average position, and run-to-run swing. Crescive does this automatically so a score reflects a true average over many answers instead of a single lucky or unlucky capture.