Definition
Fine-tuning vs RAG vs prompting is the choice of where to put what a model needs for a task. Prompting puts instructions and examples in the context window at inference time (in-context learning, Brown et al., "Language Models are Few-Shot Learners", 2020). Retrieval-augmented generation fetches relevant documents per query and puts them in the context (Lewis et al., NeurIPS 2020). Fine-tuning changes the weights with gradient updates on task examples. All three can be combined in one system.
The question comes up as soon as a base model almost works. A support bot gives generic answers, invents a refund policy, or ignores the house tone. Each failure has a different fix. Pick the wrong one and you pay for it. Fine-tuning on the policy manual will not make the bot cite this week's policy. A 4,000-token prompt of style rules costs money on every request.
The useful split is knowledge vs behaviour. Knowledge is facts the model never saw or that change: last week's prices, a private wiki, a statute amended after the training cutoff. Behaviour is how the model answers: output schema, tone, length, domain jargon, a task procedure. Knowledge belongs in context, where it can be updated and cited. Behaviour belongs in the prompt or, if the prompt cannot hold it, in the weights.
Why knowledge goes in context
Ovadia et al. ("Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", 2023) compared the two on knowledge-heavy MMLU subjects and on questions about events after the training cutoff. RAG beat unsupervised fine-tuning on the same documents in nearly every setting, for both old and new facts. Fine-tuning only began to absorb new facts when each fact appeared in many paraphrases.
Gekhman et al. ("Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?", EMNLP 2024) found a second cost. Fine-tuning examples with facts the model did not already know were learned much more slowly than familiar ones. Once they were learned, the model hallucinated more on other questions (see Hallucination). Zhou et al.'s LIMA paper (2023) states the converse as the superficial alignment hypothesis: almost all knowledge comes from pretraining, and fine-tuning mostly teaches format and style.
Retrieval has practical advantages too. Updating the index takes minutes, not a training run. Answers can cite their source passage. Access control can filter documents per user. Deleting a document removes it from future answers, while removing a fact from weights has no reliable method.
Why behaviour can go in weights
A prompt can carry a schema, a persona and a few examples (see Few-shot prompting, System prompts and role prompting and Structured output (JSON mode)). It runs out in three ways. The behaviour needs more examples than fit or than you want to pay for. Compliance is inconsistent: the model follows the style in 90% of answers, and the rest break a parser. Or the task needs a skill the base model only shows after training, such as a new output language, a domain classifier or a tool-call format. Fine-tuning on 500–5,000 good demonstrations (see Supervised fine-tuning (SFT) and Dataset curation for fine-tuning) puts the behaviour in the weights. It also lets a smaller model match a larger prompted one on a narrow task (see Knowledge distillation).
Cost: fixed vs per-request
- requests per month
- prompt and answer tokens per request; few-shot examples and retrieved chunks add to
- price per token of the base model
- price multiplier of the serving model; hosted fine-tuned endpoints are often priced above the base model
- retrieval cost per query (embedding + vector search); 0 without RAG
- training runs per month times cost per run; 0 without fine-tuning
- break-even volume above which the fine-tune is cheaper than prompting
- per-request saving; if it is the fine-tune never pays back
Fine-tuning swaps a per-request cost (long prompts) for a fixed cost (training runs). It pays off only at volume, and only if it removes enough tokens to beat the price multiplier. Every knowledge update that forces a retrain adds to . That is the economic case for keeping changing facts in an index. The engineering cost of building a dataset usually dwarfs the compute; the equation leaves it out.
| Aspect | Prompting | RAG | Fine-tuning |
|---|---|---|---|
| Changes | context (instructions, examples) | context (retrieved passages) | weights |
| Best for | anything the base model can already do | facts: fresh, private, long-tail, citable | behaviour: format, style, narrow skill |
| Update | edit the prompt, seconds | re-index, minutes | new data + training run, hours |
| Data needed | a few examples + eval set | a document corpus | hundreds to thousands of curated demonstrations |
| Per-request cost | grows with prompt length | +k chunks + retrieval call | short prompt, but m × price or own hosting |
| Main risk | prompt drift, injection | bad retrieval, context overload | stale facts, forgetting, more hallucination on new facts |
Results worth knowing
- Knowledge injection
- Ovadia et al. (2023): RAG outperformed unsupervised fine-tuning on MMLU subjects and on post-cutoff current-events questions; combining them gave no consistent gain over RAG alone.
- New facts via SFT
- Gekhman et al. (2024): unknown-fact examples are fitted late in training and raise hallucination rates; early stopping limits the damage.
- Long context
- Liu et al., "Lost in the Middle" (TACL 2024): accuracy is highest when the relevant passage sits at the start or end of a long context and drops for the middle, so stuffing everything into the prompt is not a free substitute for retrieval.
- RAFT
- Zhang et al. (2024): fine-tune on (question, retrieved documents including distractors, chain-of-thought answer quoting the right document); it improved domain RAG on PubMed, HotpotQA and Gorilla API questions over RAG with the base model.
Worked example: break-even
| Option | T_in | $ / request | 20k / month ($) | 1M / month ($) |
|---|---|---|---|---|
| 8-shot prompt | 2,400 | 1,200 + 300 µ$ = 0.00150 | 30 | 1,500 |
| RAG, k = 4 | 1,600 | 800 + 300 + 200 µ$ = 0.00130 | 26 | 1,300 |
| Fine-tune | 400 | 2 × (200 + 300) µ$ = 0.00100 | 20 + 27 = 47 | 1,000 + 27 = 1,027 |
| Fine-tune + RAG | 1,600 | 2 × 1,100 + 200 µ$ = 0.00240 | 48 + 9 = 57 | 2,400 + 9 = 2,409 |
The fine-tune saves dollars per request, so requests a month. With weekly knowledge retrains () it is . With 4 shots the prompt has and costs exactly , the same as the fine-tune, so there is no saving to repay training. Caching a long static prefix (see Prompt caching) lowers and moves further out.
1from dataclasses import dataclass23@dataclass4class Option:5 name: str6 tokens_in: int # prompt tokens per request7 tokens_out: int = 2008 price_mult: float = 1.0 # fine-tuned endpoints are often priced higher9 per_query: float = 0.0 # retrieval: embedding + vector search10 fixed: float = 0.0 # training runs per month1112 def monthly(self, n, p_in=0.5e-6, p_out=1.5e-6):13 per_req = self.price_mult * (self.tokens_in * p_in + self.tokens_out * p_out) + self.per_query14 return n * per_req + self.fixed1516train_run = 5_000 * 600 * 3 * 3e-6 # 9M training tokens at $3/M = $2717options = [18 Option("8-shot prompt", 300 + 100 + 8 * 250),19 Option("RAG, k=4", 300 + 100 + 4 * 300, per_query=2e-4),20 Option("fine-tune", 300 + 100, price_mult=2, fixed=train_run),21]22for n in (20_000, 1_000_000):23 print(n, {o.name: round(o.monthly(n), 2) for o in options})24# 20000 {'8-shot prompt': 30.0, 'RAG, k=4': 26.0, 'fine-tune': 47.0}25# 1000000 {'8-shot prompt': 1500.0, 'RAG, k=4': 1300.0, 'fine-tune': 1027.0}Combining them
A practical order of work
- 1Build an evaluation set from real queries, with expected answers or a rubric (see LLM-as-a-judge and RAG evaluation). Every later decision is measured on it.
- 2Write the best prompt: instructions, schema, a few examples. Measure. Many projects stop here.
- 3If answers lack facts or cite nothing, add retrieval (see RAG architecture). Measure retrieval recall separately from answer quality.
- 4If format, tone or a narrow skill still fails, fine-tune with LoRA on curated demonstrations, keeping the same prompt and retrieved context in the training examples that the model will see in production.
- 5Re-run the full eval after each change. Fine-tuning can quietly hurt general ability (forgetting) and the model's use of retrieved text.
Check yourself
A legal assistant must answer from a firm's 20,000 internal memos, which change daily, and must cite the memo used. The base model already writes in an acceptable style. What should you add to the prompt?
Check yourself
Using the worked-example prices, a 12-shot prompt is replaced by a fine-tune retrained once a month. What is the break-even volume?
Remember
- ✓Knowledge (fresh, private, citable facts) goes in context via retrieval; behaviour (format, style, narrow skill) goes in the prompt, or in weights when the prompt cannot hold it.
- ✓RAG beats fine-tuning for injecting facts (Ovadia et al. 2023), and fine-tuning on unknown facts raises hallucination (Gekhman et al. 2024).
- ✓Cost: ; break-even .
- ✓A fine-tune that removes too few prompt tokens to beat its price multiplier never pays back; frequent retrains push break-even out.
- ✓Combine them: prompt + RAG + a LoRA trained on retrieved context with distractors (RAFT), measured on one eval set throughout.