At a glance
| RAG | Fine-tuning | |
|---|---|---|
| Changes | The prompt (adds retrieved text) | The model's parameters |
| Best for | Missing or changing knowledge | Consistent behaviour, style, format, narrow skills |
| Updating information | Edit or re-index documents | Retrain (slow, hard to remove facts) |
| Citations / traceability | Natural: you know which passages were used | Difficult: knowledge is diffused in weights |
| Needs | Good documents, chunking, retrieval | Many high-quality examples, training pipeline |
| Typical failure | Wrong passage retrieved; model ignores it | Overfitting, forgotten abilities, stale facts |
| Per-request cost | Longer prompts (retrieved text) | Often shorter prompts; may allow a smaller model |
Neither is better in general. They answer different questions: what does the model know right now? versus how does the model behave?
What is each?
RAG retrieves relevant passages from your data when a question arrives and includes them in the prompt, so the model answers from that material.
Fine-tuning continues training a pre-trained model on your own input→output examples so it internalises a pattern: a house style, a strict output format, a classification scheme, a domain's way of reasoning.
Key differences
Knowledge vs behaviour. Fine-tuning is a poor way to reliably teach a model facts: it may absorb some, blur others, and you cannot easily update or remove them. RAG is a good fit for facts because the facts live in documents you control. Conversely, RAG is a weak way to enforce a style — you can show examples in the prompt, but a tuned model follows it more consistently and with a shorter prompt.
Freshness. Changing a policy in RAG means changing a document. In a fine-tuned model it means preparing new data and retraining.
Traceability and access control. With RAG you can show sources and restrict retrieval per user. With fine-tuning, anything in the training data is potentially retrievable by anyone who can use the model.
Effort and cost. RAG needs an indexing and retrieval pipeline, and every request pays for the retrieved tokens. Fine-tuning needs curated examples and training runs, but then may reduce prompt length per call. Costs depend on providers and scale; estimate with your own numbers.
Failure modes. RAG fails when retrieval misses or the model misuses the passage. Fine-tuning fails when data is inconsistent or too narrow. Both need evaluation (How to Evaluate AI Systems and Outputs).
When to choose which
Work through this order:
- Better prompting first. Clear instructions, examples and structure fix a surprising number of issues (Prompt Engineering: How to Write Better AI Prompts).
- Is the problem missing information? (private docs, recent events, a large knowledge base, need for citations) → RAG.
- Is the problem inconsistent behaviour? (format drifts, tone off, narrow task with abundant labelled examples, prompts getting very long) → fine-tuning.
- Both? → fine-tune for behaviour and use RAG for knowledge. Example: a model tuned to write answers in a strict support-reply format, fed with retrieved policy text.
- Small knowledge base? Putting it directly in the prompt may beat both.
Example scenarios
- "Answer questions about our 400-page handbook, with page references." → RAG.
- "Always output tickets as this 12-field JSON, in our classification scheme, from messy emails." → prompting with examples first; fine-tune if reliability or cost requires it.
- "Chatbot that knows this week's prices and speaks in our brand voice." → RAG for prices, prompting or fine-tuning for voice.
- "Make the model know who our new CEO is." → RAG or a simple prompt fact, not fine-tuning.
Common mistakes
- Fine-tuning to add facts and expecting reliable recall.
- Building RAG when the real issue is output format.
- Comparing without a baseline and test set.
- Assuming either removes hallucinations.
- Ignoring maintenance: RAG indexes need refreshing; tuned models need re-evaluating when the base model changes.