The Real Reason Your RAG Pipeline Keeps Hallucinating

You know the ritual. The pipeline hallucinated in front of a customer, so you swapped the embedding model. Then you upgraded the LLM. Then you added a system prompt that says, in increasingly desperate capital letters, ONLY USE THE PROVIDED CONTEXT. And this morning it confidently cited a policy document that doesn’t exist.

You’re in good company. When Stanford’s RegLab and HAI researchers audited the AI legal research tools sold by LexisNexis and Thomson Reuters, products marketed as “hallucination-free” thanks to retrieval-augmented generation, they found hallucination rates between 17% and 34% on pre-registered legal queries. RAG genuinely helped: raw GPT-4 hallucinated far more. But the gap between “reduced” and “eliminated” is where production systems live, and that gap has a structure.

Hallucination in a RAG pipeline is rarely one failure. It’s three, conspiring: retrieval fails quietly, the model was trained to answer anyway, and nothing in the pipeline measures the space between those two facts. Swapping models addresses none of them.

Conspirator one: retrieval fails more often than you think

The freshest evidence here is also the most uncomfortable. In November 2025, a team including 18 medical experts produced 80,502 annotations across 800 RAG outputs on real patient and USMLE-style queries. Standard RAG didn’t just underdeliver; it degraded factuality by up to 6% and completeness by 5% compared with the same models running without retrieval. The root cause sat upstream of the LLM entirely: only 22% of the top-16 retrieved passages were relevant to the query.

Before you dismiss that as a hard-domain problem, Anthropic measured retrieval failure on clean, curated corpora while developing its contextual retrieval technique and found standard embedding search failed to surface the needed chunk in the top 20 results 5.7% of the time. One query in eighteen, on well-groomed data, with nothing exotic going on.

This is why the canonical engineering taxonomy of RAG failures, Barnett et al.’s seven failure points, matters so much: three of the seven (missing content, missed top-ranked documents, and context consolidation failures) occur before the language model generates a single token. Unite.AI has published a thorough walkthrough of that taxonomy and the evaluation frameworks that map to it, so I won’t rebuild it here. The point this article adds is about incentives: embedding similarity is a proxy for relevance, not a guarantee of it, and recent theoretical work from Google DeepMind indicates single-vector embeddings have hard mathematical limits on which combinations of relevant documents they can represent at all. A retriever that returns plausible-but-wrong chunks is doing exactly what cosine similarity does.

Conspirator two: the model was trained to guess

Now hand that flawed context to a language model, and ask what the model’s training taught it to do with gaps.

OpenAI answered this directly in its September 2025 paper Why Language Models Hallucinate: standard training and evaluation reward guessing over acknowledging uncertainty. Benchmarks score binary accuracy, abstaining scores zero, and so, like students facing a multiple-choice exam, models learn that a confident guess beats a blank. The paper’s own comparison makes it concrete: on SimpleQA, one reasoning model abstained 1% of the time and was wrong 75% of the time, while a differently tuned sibling abstained 52% of the time and cut its error rate to 26%.

The RAG-specific benchmarks show what that incentive does inside a pipeline. The RGB benchmark tested whether models refuse to answer when handed only irrelevant documents. The best negative-rejection rate across every model tested was 45%, meaning that even the best model, given nothing but junk context, answered anyway more than half the time. ClashEval found the mirror-image failure: when retrieved content contradicted knowledge the model already had right, models abandoned their own correct answer in favor of the wrong context more than 60% of the time. Faithfulness fails in both directions, and Salesforce’s FaithEval adds the unsettling coda that bigger models are not reliably more faithful.

Anthropic’s interpretability team has even traced the mechanism. In their analysis, refusal is the model’s default circuit; a “known entity” feature suppresses that refusal when the model recognizes something. Hallucination happens when the feature misfires, when the model recognizes the shape of your question, lacks the substance, and confabulates fluently into the gap.

And if you’re hoping the frontier will simply outgrow this: Vectara’s hallucination leaderboard measures something far easier than RAG, summarizing a single document placed directly in front of the model, and as of its May 2026 update, GPT-4o still fabricated in 9.6% of summaries and Claude Opus 4 in 12%. Even with retrieval solved perfectly, generation leaks.

The instinctive fix makes it worse

Faced with all this, most teams reach for volume. Retrieve more chunks. Buy the bigger context window. Stuff in everything that might help and let the model sort it out.

The evidence runs hard the other way. Chroma’s context rot study tested 18 models, including GPT-4.1, Claude 4, and Gemini 2.5, and found performance growing “increasingly unreliable as input length grows,” with a single distractor document measurably cutting accuracy and four distractors cutting it substantially. The earlier Lost in the Middle research found the now-famous U-curve: information buried mid-context gets ignored even by long-context models. And the medical audit above is what those dynamics look like end to end, a system retrieving sixteen passages of which twelve-plus are irrelevant, then handing the pile to a model trained never to say “I don’t know.”

More retrieval without more precision just manufactures distractors. Precision beats recall in grounded generation, and it isn’t close.

Conspirator three: nobody is measuring the gap

Barnett et al. put the operational truth in one line: “validation of a RAG system is only feasible during operation.” You cannot sign off a RAG pipeline in staging, because its failure modes are a joint property of your corpus, your queries, and your users, none of which hold still.

Yet most production pipelines track end-to-end answer quality at best, which conflates the two conspirators above into a single unexplainable number. The standard decomposition, sometimes called the RAG triad, separates context relevance (did retrieval find the right material?), groundedness (does the answer stick to that material?), and answer relevance (does it address the question?). Frameworks like RAGAS and TruLens implement it. I’ll be honest that no rigorous survey exists quantifying how few production teams run these; what exists is the practitioner record, and it mostly describes evaluation by vibes. If your team has no dashboard distinguishing retrieval failures from faithfulness failures, every hallucination will keep looking like a model problem, and you’ll keep buying model-shaped fixes.

What actually works, in order

Measure retrieval separately from generation. The unglamorous fix everyone skips, and in my view the highest-leverage item on this list, because it converts an argument about which model to buy into a diagnosis. If context relevance is bad, no generator can save you. If groundedness is bad with good context, no retriever can.

Build the precision stack. Anthropic’s published numbers are the cleanest demonstration of stacked retrieval fixes anywhere: contextual chunk embeddings cut top-20 retrieval failure from 5.7% to 3.7%, adding BM25 hybrid search (classic keyword matching alongside vector search) brought it to 2.9%, and adding a reranker, a second-stage model that re-scores the candidate chunks for actual relevance, reached 1.9%, a 67% total reduction. Unite.AI has covered why two-stage retrieval punches above its weight in detail.

Reward abstention. OpenAI’s prescription is to penalize confident errors more than expressed uncertainty, and you can implement the local version today: prompt for and evaluate “I don’t know” responses, and add a grounding check, an entailment model (NLI) that verifies each generated claim is supported by the retrieved text before the answer ships. The RAGTruth work showed a small fine-tuned detector can match GPT-4-based prompting at catching unsupported claims.

Rewrite queries and filter evidence. In the medical audit, query reformulation plus evidence filtering recovered up to 12 points of factuality. Cheap, boring, effective.

Consider agentic retrieval last. Multi-step retrieval that reasons about what to fetch next (primer here) genuinely helps hard multi-hop queries, but it adds latency, cost, and new failure modes. It’s a capstone, not a foundation.

The model was the least broken part

Look back at the ritual from the opening. Every step of it, the embedding swap, the model upgrade, the shouting system prompt, treated hallucination as a defect in the generator. The evidence says the generator was doing what its training rewards, with the materials your retriever handed it, unobserved by any metric that could have said which of those was failing.

Your RAG pipeline isn’t broken. It’s obedient. It does exactly what its incentives reward, and until you measure retrieval and generation separately and make “I don’t know” a scoring category instead of a failure, those incentives say: guess.