Fine-tuning vs RAG vs prompting: which one should you use?
Three tools improve an LLM app, and they solve different problems. Pick the cheapest one that fixes your actual failure.
When an LLM feature is not good enough, people often jump to fine-tuning. Usually that is the wrong first step. There are three main levers, and each fixes a different kind of problem.
1. Prompting
You change the instructions and examples you send with each request.
- Fixes: unclear task, wrong format, wrong tone, missing rules.
- Cost: almost nothing. Instant to change.
- Limit: the model cannot use facts it has never seen, and very long prompts cost more per call.
Always try this first. A clear prompt with a few good examples solves a surprising number of problems.
2. RAG (retrieval-augmented generation)
You fetch relevant documents at question time and put them in the prompt.
- Fixes: missing or changing knowledge, private data, the need for citations, hallucinated facts.
- Cost: you build and maintain a search pipeline. Each call uses more tokens.
- Limit: quality depends on retrieval, and it does not change how the model behaves or writes.
Use it when the problem is "the model does not know this".
3. Fine-tuning
You train the model further on your own examples so it changes its behaviour.
- Fixes: a consistent style or format that prompting cannot hold, a narrow specialised task, shorter prompts at scale, or making a small cheap model good at one job.
- Cost: you need a good dataset (often hundreds to thousands of high-quality examples), training time and money, evaluation, and re-training when things change.
- Limit: it is poor at reliably teaching new facts, can make the model worse at other things, and is slow to update. Not every model or provider supports it.
Techniques such as LoRA and other parameter-efficient methods train a small add-on rather than the whole model, which lowers cost.
Use it when the problem is "the model does not behave the way I need, even with good prompts".
A simple decision guide
| Your problem | Try first |
|---|---|
| Wrong format, tone, or misunderstood task | Better prompt, examples, structured output |
| Needs your documents or fresh facts | RAG |
| Needs citations or an audit trail | RAG |
| Must follow a strict house style across thousands of outputs | Fine-tuning (after prompting) |
| Narrow, high-volume task and you want a small cheap model | Fine-tuning or distillation |
| Reasoning is weak | A stronger model, or step-by-step prompting |
| Needs exact calculations or live data | Tools (function calling), not training |
A sensible order
- Define success with a test set of real examples and a way to score them.
- Improve the prompt and try a stronger model.
- Add RAG if knowledge is the gap.
- Consider fine-tuning only if a measured gap remains and you have good data.
- Combine them if needed. Fine-tuned models can still use RAG.
Without step 1 you cannot tell whether any change helped.
Data matters most for fine-tuning
- Examples must be correct, consistent and representative of real inputs.
- Poor or biased data teaches poor behaviour.
- Keep a separate evaluation set the model never trains on.
- Remove personal or secret data unless you have the right to use it and strong controls.
- Check the provider's terms for data use and retention.
Cost and maintenance thinking
- Prompting and RAG change instantly. Fine-tuned models must be re-trained when requirements or base models change.
- Fine-tuning can reduce per-request cost by shortening prompts or using a smaller model, but the up-front effort is real. Do the sums with your own volume.
- New base models improve quickly, so a fine-tune of last year's model may be beaten by a better base model with a good prompt.
Common mistakes
- Fine-tuning to add knowledge, then being surprised by wrong facts.
- Skipping evaluation, so improvements are guesses.
- Using a huge prompt of rules when a few examples would work better.
- Building RAG when a bigger context window and a handful of documents would do.
Takeaway: prompting for instructions, RAG for knowledge, fine-tuning for behaviour. Measure first, try the cheapest lever, and escalate only when your tests show a real gap.
AI & LLM series, Part 2 (quality, agents and cost): 1 of 5.
