RAG or Fine-Tuning? What Real Client Projects Taught Us About Choosing

Almost every enterprise AI conversation I have reaches the same fork within the first hour. The client has a large language model that is impressive in a demo but does not know their products, their policies or their way of writing. Someone in the room asks: should we fine-tune the model on our data, or should we build retrieval-augmented generation (RAG) around it?

The honest answer is that the question is usually framed wrong. RAG and fine-tuning solve different problems. Teams that treat them as competing options tend to spend months and a lot of GPU budget fixing the wrong thing. Here is how we decide in practice, and what we have seen go right and wrong.

Start With What the Model Is Missing

When a model gives a poor answer on a business task, it is missing one of two things.

It is missing knowledge. It does not know your latest price list, the clause in your 2025 supplier contract or the troubleshooting steps for a product that launched last month. This is a facts problem.

It is missing behaviour. It knows enough, but it does not respond the way you need. It ignores your output format, writes in the wrong tone, misreads your internal codes or cannot follow a narrow, repeated task such as classifying tickets into your 40 categories. This is a skills problem.

RAG is the tool for the first problem. Fine-tuning is the tool for the second. Most of the confusion I see comes from trying to use fine-tuning to teach facts.

Why Fine-Tuning Is a Poor Way to Add Knowledge

It is tempting to think that if you train a model on your documents, it will “know” them. In practice, it learns the style of those documents much more reliably than the facts inside them. Researchers at Microsoft compared the two approaches directly for knowledge-heavy questions and found that RAG consistently outperformed unsupervised fine-tuning, both for facts the model had seen during training and for entirely new ones.

There are practical problems too. Facts change. Every time your policy or catalogue changes, a fine-tuned model is out of date until you train it again. You also lose traceability: when a fine-tuned model states a figure, you cannot point to the document it came from. For regulated clients in insurance and healthcare, that alone usually ends the discussion.

RAG, first described in the 2020 paper by Lewis et al., keeps knowledge outside the model. You retrieve the relevant passages at question time and pass them to the model as context. Updating knowledge means updating an index, not retraining. And every answer can cite its source.

A logistics client of ours is a good example. Its customers wanted instant answers about where their shipments were and which documents were missing. That information changes every few minutes. No amount of training would have kept a model current, so the assistant we built answers from live tracking and IoT data at question time instead of from memory. Retrieval was the only design that could work.

Where Fine-Tuning Earns Its Place

Fine-tuning is not dead. It is just narrower than people expect. We reach for it in three situations:

• Strict, repeated output formats. When a model must produce the same structured output thousands of times a day, such as extracting fields from invoices into a fixed JSON schema, a fine-tuned small model is often more reliable and far cheaper than prompting a large one.
• Domain language. Some fields have shorthand the base model misreads. Logistics status codes, clinical abbreviations and internal product codes are common examples. A modest fine-tune can fix this.
• Cost and latency at scale. Once a task is well defined, a fine-tuned small model can replace a large general model. Methods such as LoRA make this affordable, because you train a small set of extra weights rather than the whole model.

Notice that none of these is about facts. They are about teaching a model how to behave on a task you already understand well.

Most RAG Failures Are Retrieval Failures

When a team tells me “we tried RAG and it did not work”, the model is rarely the problem. The retrieval step is. The model can only be as good as the passages it is given.

The fixes are unglamorous but effective:

• Chunk by meaning, not by character count. Splitting a contract every 500 characters cuts clauses in half. Splitting by section and heading keeps context intact.
• Add context to each chunk. A chunk that says “the limit is 30 days” is useless without knowing which policy it belongs to. Anthropic reported that adding short document context to each chunk before indexing, an approach it calls contextual retrieval, reduced failed retrievals by 49%, and by 67% when combined with reranking.
• Combine keyword and vector search. Vector search is good at meaning but weak on exact terms such as part numbers. Hybrid search covers both.
• Test retrieval separately. Build a set of real questions with known correct source passages, and measure whether the right passage appears in the top results before you judge the final answers.

When to Use Both

In production, the strongest systems often combine the two. RAG supplies the facts. A lightly fine-tuned model handles the format, tone and domain language. Research is moving in the same direction: the RAFT approach from UC Berkeley fine-tunes a model specifically to make better use of retrieved documents and ignore irrelevant ones.

But order matters. We almost always build RAG first, measure where it falls short, and only then decide whether fine-tuning is worth it. Starting with fine-tuning means paying for training before you know what the real gap is.

A Simple Decision Checklist

1. Does the task depend on facts that change more than once a quarter? Use RAG.
2. Do users or auditors need to see where an answer came from? Use RAG.
3. Is the problem the format, tone or vocabulary of answers, not their facts? Consider fine-tuning.
4. Is it a narrow task run at high volume where cost or speed matters? Consider fine-tuning a smaller model.
5. Have you measured retrieval quality on its own? If not, do that before anything else.

The Takeaway

RAG and fine-tuning are not rivals. One gives a model the right information, the other shapes how it behaves. Get clear on which problem you actually have, fix retrieval before blaming the model, and treat fine-tuning as an optimisation you earn once the system works, not a shortcut to make it work.