What is the difference between RAG and fine-tuning?

Retrieval-augmented generation gives a model your documents at question time, so it answers from information it has just been handed. Fine-tuning adjusts the model's own weights by training on examples, changing how it behaves rather than what it knows. RAG teaches knowledge; fine-tuning teaches behaviour, format and tone.

Should I use RAG or fine-tuning for a company knowledge base?

RAG, almost always. Knowledge bases change, and retrieval reflects an edit the moment it is saved, while a fine-tuned model would need retraining. Retrieval also lets the answer cite its source document, which matters when someone needs to verify what the system said.

When is fine-tuning actually the right choice?

When you need consistent behaviour rather than current facts: a specific output format the model keeps deviating from, a distinctive tone, a specialised classification task with plenty of labelled examples, or shorter prompts and lower latency at high volume once behaviour is settled.

Is fine-tuning more accurate than RAG?

Not for factual questions about your own content. Fine-tuning teaches patterns, not reliable fact recall, and a fine-tuned model can still state outdated information confidently because nothing grounded it. For questions with a right answer that lives in your documents, retrieval is both more accurate and auditable.

What should I do before choosing either?

Build an evaluation set: fifty to a hundred real questions with known-good answers. Without it you cannot tell whether a change helped, and most disappointing AI projects are ones where quality was a matter of opinion rather than a number.

RAG vs fine-tuning: which does your business need?

These get framed as competing options. They are not — they solve different problems, and the one most businesses ask for is usually not the one they need. A decision guide, with the honest answer that neither is often the fix.

Short answer

RAG changes what the model knows; fine-tuning changes how it behaves. If the complaint is that the model does not know things about your business, it is RAG and was never a choice. Fine-tuning is for format, tone and narrow tasks — and most disappointing results are retrieval quality, not the model.

This question usually arrives already framed as a choice between two options, and often with a preference attached — fine-tuning sounds more serious, more bespoke, more like real engineering. It is worth resetting the framing, because they are not alternatives to each other:

RAG changes what the model knows. Fine-tuning changes how the model behaves.

Once you see it that way, most cases answer themselves.

What each one actually is

Retrieval-augmented generation adds a step before the model answers. The question comes in, the system searches your own content, pulls the most relevant passages, and hands them to the model along with the question. The model is not remembering your refund policy; it is reading it, just now. Change the document and the next answer changes with it — and the answer can cite which document it came from.

Fine-tuning adjusts the model's weights by training it on examples of input and desired output. You are not giving it facts to look up; you are shifting its default behaviour, so it formats things your way, adopts a tone, or handles a narrow classification task the way your examples do.

	RAG	Fine-tuning
Teaches	Knowledge — what is true in your business	Behaviour — format, tone, task
Updating it	Edit the document; effective immediately	Retrain on new examples
Can cite sources	Yes	No
Needs	Documents worth retrieving	Hundreds to thousands of good examples
Cost shape	Ongoing retrieval and longer prompts	Training cost up front, cheaper prompts after
Fails by	Retrieving the wrong passage	Confidently stating stale facts

The decision, in one pass

Ask what is actually wrong with the default model's output:

It does not know things about us. → RAG. Nothing else fixes this, and fine-tuning is a poor substitute because it teaches patterns rather than reliable recall.
It knows enough but answers in the wrong shape — wrong format, wrong register, ignores your conventions. → Try prompting first. If it still drifts across hundreds of runs, fine-tune.
It is too slow or too expensive at volume, and behaviour has settled. → Fine-tuning can shorten prompts and cut latency. This is an optimisation, and it should come last.
Both knowledge and behaviour. → RAG first, always. Get grounding right, then fine-tune the behaviour on top if it is still worth it. Usually it is not.

The answer nobody wants: it is probably your retrieval

When a document-grounded assistant disappoints, the model is rarely the culprit. The retrieval step is. Common causes, in the order we tend to find them:

Documents chunked badly — split mid-table or mid-clause, so the retrieved passage is missing the half that mattered.
Pure vector search where keywords were needed. Semantic search is poor at exact identifiers — part numbers, error codes, policy references. Hybrid search fixes more real complaints than any model upgrade.
No reranking. Retrieving twenty candidates and reranking to the best five is one of the cheapest quality wins available.
The source material is genuinely bad. If the policy is ambiguous in the document, the assistant will be ambiguous about it. No amount of AI resolves a contradiction your own documentation contains.

That last one is worth sitting with. A retrieval system is a mirror held up to your documentation, and it frequently reveals that the documentation was the problem all along.

Do this before either

Build an evaluation set: fifty to a hundred real questions with known-good answers, from people who actually ask them. Run it on every change.

Without one, "is it better now?" is a matter of opinion, and every change becomes an argument. With one, it is a number. This is the single largest predictor we see of whether an AI project succeeds — not the model, not the technique, but whether anyone can tell the difference between a good version and a bad one. It is why we agree a metric before starting AI feature work at all.

Where agents come into it

Both techniques are about producing better answers. If what you actually need is for something to be done — a ticket triaged, a record updated — that is a different architecture with different hard parts, and we covered the distinction in agents versus chatbots. Retrieval quality still matters there; it just stops being the whole problem.

The short version: RAG for knowledge, fine-tuning for behaviour. If the complaint is "it doesn't know our stuff," it is RAG and it was never a choice. If quality is disappointing, suspect chunking, hybrid search and reranking long before you suspect the model — and build an evaluation set first, so you can tell whether anything you change actually helped.

Not sure which one your problem is?

Book a free 30-minute discovery call. We build AI features on Claude and other frontier models, with a real evaluation set — and we will tell you when the honest answer is "fix the documentation first."

Book a free discovery call

RAG vs fine-tuning: which does your business actually need?

What each one actually is

The decision, in one pass

The answer nobody wants: it is probably your retrieval

Do this before either

Where agents come into it