Thought Leaders
When AI Engines Agree, That Is Not Proof: What 300 Product-Fact Queries Show

Key PointsAgreement between models measures how widely a version has spread rather than whether…
View more
- Agreement between models measures how widely a version has spread rather than whether it is current.
- Cross-checking catches one-engine outliers but cannot detect an answer that all models share.
- An answer can contain every official note while placing those notes in the wrong tiers.
This article is based on findings from Scento’s October 2026 study of how ChatGPT, Gemini and Perplexity answer questions about perfume notes (full method, data and limitations).
A common safety net in AI agent design is to ask more than one model, or the same model more than once, and trust the answer they agree on. The idea has solid roots. Self-consistency, which samples several reasoning paths and keeps the majority answer, improves accuracy on reasoning benchmarks. Using one model to grade another, studied in Judging LLM-as-a-Judge, reaches high agreement with human raters, although the authors also document position, verbosity and self-enhancement biases.
Product facts are a good place to test whether agreement means accuracy, because the right answer is usually written down by the company that makes the product. We tested it on perfume, and the answer was: only partly.
How the Test Worked
The prompt was a single sentence asking for a fragrance’s top, heart and base notes. We sent it for each of 100 perfumes, drawn from 45 houses, to ChatGPT, Gemini and Perplexity on 20 October 2026. All calls ran once, in English, from a US location: 300 in total. Gemini returned 99 answers, as one call ended in a service error.
For 91 perfumes, the reference was the note list the house publishes on its own site. For the remaining 9, it was a list two authorised retailers agree on. Grading was done by a script that used no language model at all. It mapped every note in each answer onto a 411-note glossary, then compared the result with the reference. When we checked 20 gradings by hand, 17 agreed exactly and 19 within one note. That check was not blind, because the reviewer could see the script’s result.
We measured four things: recall (how many official notes an answer contains), precision (how many of the notes it names are official), exact match (all official notes, nothing extra) and placement (whether each note sits in the right top, heart or base tier).
Finding 1: The Engines Agree on What Is There
On recall, the three engines look almost interchangeable. Median recall was 1.00 for each. ChatGPT returned every official note in 82% of answers, Perplexity in 78% and Gemini in 69%.
Agreement on the full answer was much rarer. Of the 99 perfumes all three engines answered, all three returned every official note for 59. All three matched the official list exactly for only 39. At least one engine was exact for 79, and for 20 perfumes none was.
In about one case in five (20 of the 99 perfumes), all three engines returned every official note, yet at least one of them also added notes that are not on the list. A recall check would have passed all three answers.
Finding 2: Shared Answers Pass Every Consensus Check
The most instructive results were the ones the engines had in common.
For Jean Paul Gaultier Le Male Elixir, the two agreeing retailers give three notes: lavender on top, tonka bean in the heart, benzoin in the base. ChatGPT, Gemini and Perplexity each produced an identical seven-note pyramid. Each included all three reference notes, put tonka bean and benzoin in each other’s tiers, and added four more. Three separate engines, one identical answer, and that answer departs from the reference in the same places.
For Viktor&Rolf Flowerbomb, the house lists four notes. Gemini and Perplexity both returned the same 11-note pyramid, missed one official note and added eight. Perplexity’s answer then noted that the brand’s current description highlights the four official notes, so it held both versions and led with the older one.
Our data cannot say why the answers match. But the practical point does not depend on the cause. When systems draw on the same pool of web text, a majority vote measures how widely a version has spread, not whether it is current. Agreement between them is weak evidence that the answer matches what the brand itself publishes today.
Finding 3: One-Engine Outliers Are the Easy Ones
Other extra notes belonged to one engine alone, and these are exactly what comparison catches. In 32% of its answers Gemini added at least one note that neither of the other engines named, against 10% for ChatGPT and 11% for Perplexity.
For Prada Infusion de Vétiver, the house lists four notes. ChatGPT and Perplexity returned exactly those four, in exactly the right places. Gemini returned twelve, including eight that no other engine gave, for a precision of 0.33. For Rochas Eau de Rochas, Perplexity matched the ten-note list exactly, while Gemini returned 18 notes, ten of them named by no other engine.
Cross-checking works well here: an outlier stands out against two agreeing answers. That is the asymmetry builders should keep in mind. Disagreement is a useful alarm. Agreement is not a certificate.
Finding 4: Right Facts, Wrong Structure
Some answers had the right notes in the wrong places. For Montblanc Explorer, ChatGPT matched six of the nine official notes, but only two of those six sat in the correct tier, a placement accuracy of 33%. A check that only asks whether a note appears somewhere in the answer would not see that four of the six are in the wrong tier.
For any agent that turns product facts into decisions, structure carries meaning. The same is true of sizes, compatibility, dosage or ingredients in other categories. An evaluation that checks only whether the facts appear somewhere in the answer will miss this class of mistake.
A Pattern by Product Type
Answers about designer perfumes carried off-list notes more often than answers about niche ones on all three engines, with gaps of 22 to 32 percentage points. ChatGPT showed the widest split: four in ten of its designer answers included at least one off-list note, compared with fewer than one in ten of its niche answers.
A short official list gives an engine more room to add notes, so we controlled for list length by pooling all three engines and sorting answers into three bands. Designer answers still carried extras more often in each one: 70% against 31% for lists of five notes or fewer (27 and 29 answers), 61% against 21% for six to eight notes (51 and 42), and 26% against 13% for nine or more (72 and 77). The samples are small, and because each group came from its own set of houses and source pages, we cannot isolate the cause. What the pattern does suggest is that accuracy varies by product segment, and a single figure for a whole catalogue can hide that.
What This Means for Teams Building Shopping Agents
Use consensus to find disagreement, not to confirm truth. Comparing models is a cheap way to flag one-off additions. It cannot detect an answer the models share, so it should not be the last check.
Ground owned facts in an owned source. Where a fact has an owner, such as a manufacturer’s specification, the most reliable path is to retrieve it from a structured record that carries its source page. Open standards make this practical: schema.org Product markup on the page, and the Model Context Protocol for letting agents query a data source directly. Google’s guidance on AI features says structured data should match the visible text on the page, and that no special schema.org markup is needed to appear in those features.
Score precision, exact match and structure, not just recall. An answer that contains every fact plus three invented ones passes a recall test and still misleads the user.
Grade deterministically where reference data exists. A rule-based grader compared with structured data is repeatable and auditable, and it does not share a model’s blind spots.
Run more than once. Answers vary from run to run, a point Andrew Lissimore made on this site about measuring AI shopping recommendations. Our study made one call per perfume per engine, so its run-to-run variation is unknown.
Limits of the Study
Each perfume was asked once per engine, on one day, in English, from one country. Claude and engines without web search were not tested. A note missing from an official list is not automatically wrong, because houses sometimes publish only key notes, so precision is a lower bound on correctness. The grader was tuned on the same 100 perfumes, and the sample is mostly Western designer and niche houses whose lists could be verified.
Within those limits the pattern was consistent. Three engines agreeing tells you that a version of the facts is widely repeated. Only the source tells you whether it is current. So here is the question for anyone whose agent checks itself against a second model: what does your agent do when two models give the same wrong answer?












