Fine-tuning vs RAG vs prompting

Prompt first; add RAG when the model lacks facts (fresh, private, citable), fine-tune when it lacks behaviour (format, style, a narrow skill). A fine-tune saves money only if its shorter prompts repay training.

8 min read961 wordscore0 viewsupdated 2026-10-21

Definition

Fine-tuning vs RAG vs prompting is the choice of where to put what a model needs for a task. Prompting puts instructions and examples in the context window at inference time (in-context learning, Brown et al., "Language Models are Few-Shot Learners", 2020). Retrieval-augmented generation fetches relevant documents per query and puts them in the context (Lewis et al., NeurIPS 2020). Fine-tuning changes the weights with gradient updates on task examples. All three can be combined in one system.

The question comes up as soon as a base model almost works. A support bot gives generic answers, invents a refund policy, or ignores the house tone. Each failure has a different fix. Pick the wrong one and you pay for it. Fine-tuning on the policy manual will not make the bot cite this week's policy. A 4,000-token prompt of style rules costs money on every request.

The useful split is knowledge vs behaviour. Knowledge is facts the model never saw or that change: last week's prices, a private wiki, a statute amended after the training cutoff. Behaviour is how the model answers: output schema, tone, length, domain jargon, a task procedure. Knowledge belongs in context, where it can be updated and cited. Behaviour belongs in the prompt or, if the prompt cannot hold it, in the weights.

1. best prompt + eval set (50–200 cases)zero/few-shot, system prompt, output schemapasses the eval?yesship promptingno: what fails?missing knowledgefacts past cutoff, privatechanges often, cite sourcesRAGtop-k chunks into the promptupdate: re-index, minutescost: +k × 300 tok/requestwrong behaviourformat, tone, house stylea skill prompts can't elicitfine-tune (SFT / LoRA)500–5,000 demonstrationsupdate: retrain, hourscost: training + m × pricebothdomain facts + house stylee.g. support or legal QARAG + fine-tunetune on (query, docs, answer)with distractor docs: RAFTfacts in index, style in WKnowledge goes in context (retrieval); behaviour goes in weights (fine-tuning).The prompt stays in every option; re-run the same eval set after each change.
fig.The decision flow. Start with the best prompt you can write and an evaluation set of 50–200 real cases. If it fails, classify the failure: missing knowledge leads to RAG, wrong behaviour to fine-tuning (usually LoRA), and both to a fine-tuned model that is trained to read retrieved documents (RAFT). The prompt stays in every option.

Why knowledge goes in context

Ovadia et al. ("Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", 2023) compared the two on knowledge-heavy MMLU subjects and on questions about events after the training cutoff. RAG beat unsupervised fine-tuning on the same documents in nearly every setting, for both old and new facts. Fine-tuning only began to absorb new facts when each fact appeared in many paraphrases.

Gekhman et al. ("Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?", EMNLP 2024) found a second cost. Fine-tuning examples with facts the model did not already know were learned much more slowly than familiar ones. Once they were learned, the model hallucinated more on other questions (see Hallucination). Zhou et al.'s LIMA paper (2023) states the converse as the superficial alignment hypothesis: almost all knowledge comes from pretraining, and fine-tuning mostly teaches format and style.

Retrieval has practical advantages too. Updating the index takes minutes, not a training run. Answers can cite their source passage. Access control can filter documents per user. Deleting a document removes it from future answers, while removing a fact from weights has no reliable method.

Why behaviour can go in weights

A prompt can carry a schema, a persona and a few examples (see Few-shot prompting, System prompts and role prompting and Structured output (JSON mode)). It runs out in three ways. The behaviour needs more examples than fit or than you want to pay for. Compliance is inconsistent: the model follows the style in 90% of answers, and the rest break a parser. Or the task needs a skill the base model only shows after training, such as a new output language, a domain classifier or a tool-call format. Fine-tuning on 500–5,000 good demonstrations (see Supervised fine-tuning (SFT) and Dataset curation for fine-tuning) puts the behaviour in the weights. It also lets a smaller model match a larger prompted one on a narrow task (see Knowledge distillation).

Cost: fixed vs per-request

C(N)=N[m (Tin pin+Tout pout)+r]+U CtrainC(N)=N\big[m\,(T_\text{in}\,p_\text{in}+T_\text{out}\,p_\text{out})+r\big]+U\,C_\text{train}
NN
requests per month
Tin,ToutT_\text{in},T_\text{out}
prompt and answer tokens per request; few-shot examples and retrieved chunks add to TinT_\text{in}
pin,poutp_\text{in},p_\text{out}
price per token of the base model
mm
price multiplier of the serving model; hosted fine-tuned endpoints are often priced above the base model
rr
retrieval cost per query (embedding + vector search); 0 without RAG
U CtrainU\,C_\text{train}
training runs per month times cost per run; 0 without fine-tuning
N∗=U Ctraincprompt−cft,c=m (Tin pin+Tout pout)+rN^\ast=\frac{U\,C_\text{train}}{c_\text{prompt}-c_\text{ft}},\qquad c=m\,(T_\text{in}\,p_\text{in}+T_\text{out}\,p_\text{out})+r
N∗N^\ast
break-even volume above which the fine-tune is cheaper than prompting
cprompt−cftc_\text{prompt}-c_\text{ft}
per-request saving; if it is ≤0\le 0 the fine-tune never pays back

Fine-tuning swaps a per-request cost (long prompts) for a fixed cost (training runs). It pays off only at volume, and only if it removes enough tokens to beat the price multiplier. Every knowledge update that forces a retrain adds to UU. That is the economic case for keeping changing facts in an index. The engineering cost of building a dataset usually dwarfs the compute; the equation leaves it out.

AspectPromptingRAGFine-tuning
Changescontext (instructions, examples)context (retrieved passages)weights
Best foranything the base model can already dofacts: fresh, private, long-tail, citablebehaviour: format, style, narrow skill
Updateedit the prompt, secondsre-index, minutesnew data + training run, hours
Data neededa few examples + eval seta document corpushundreds to thousands of curated demonstrations
Per-request costgrows with prompt length+k chunks + retrieval callshort prompt, but m × price or own hosting
Main riskprompt drift, injectionbad retrieval, context overloadstale facts, forgetting, more hallucination on new facts
The three options compared

Results worth knowing

Knowledge injection
Ovadia et al. (2023): RAG outperformed unsupervised fine-tuning on MMLU subjects and on post-cutoff current-events questions; combining them gave no consistent gain over RAG alone.
New facts via SFT
Gekhman et al. (2024): unknown-fact examples are fitted late in training and raise hallucination rates; early stopping limits the damage.
Long context
Liu et al., "Lost in the Middle" (TACL 2024): accuracy is highest when the relevant passage sits at the start or end of a long context and drops for the middle, so stuffing everything into the prompt is not a free substitute for retrieval.
RAFT
Zhang et al. (2024): fine-tune on (question, retrieved documents including distractors, chain-of-thought answer quoting the right document); it improved domain RAG on PubMed, HotpotQA and Gorilla API questions over RAG with the base model.
monthly cost = N · m · (tokens_in · p_in + tokens_out · p_out) + retraining1k10k100k1M10M$1.00$10$100$1,000$10,000$100,000requests per month N (log)$ per month (log); dashed = fails or not neededbreak-even 54kat N = 20k requests / monthfew-shot prompting$30 · $1.50/1k reqRAG (not needed)$26 · $1.30/1k reqfine-tune$47 · $1.00/1k req + $27fine-tune + RAG (not needed)$57 · $2.40/1k req + $9.00cheapest that meets requirementsfew-shot prompting: $30fine-tune cheaper than promptingabove 54k requests/monthassumed: p_in $0.50/M, p_out $1.50/M, fine-tuned model m = 2×, retrieval $0.20 per 1k queries,one training run $27 (9M tokens); FT + RAG retrains once a quarter; engineering and labelling excluded
requirements
fig.Monthly cost of each option under the cost equation, with illustrative prices ($0.50/M input, $1.50/M output, fine-tuned model at 2×, $27 per training run). Defaults: 20,000 requests, 8 shots, k = 4, one retrain a month, no requirement. Prompting costs $30 and the fine-tune $47; break-even is 54,000 requests. At 1M requests the fine-tune costs $1,027 against $1,500. Four updates a month push break-even to 216,000. At 4 shots or fewer the fine-tune never pays back. Toggle 'facts the model lacks' and RAG wins at $26; toggle both and only fine-tune + RAG remains ($57).

Worked example: break-even

OptionT_in$ / request20k / month ($)1M / month ($)
8-shot prompt2,4001,200 + 300 µ$ = 0.00150301,500
RAG, k = 41,600800 + 300 + 200 µ$ = 0.00130261,300
Fine-tune4002 × (200 + 300) µ$ = 0.0010020 + 27 = 471,000 + 27 = 1,027
Fine-tune + RAG1,6002 × 1,100 + 200 µ$ = 0.0024048 + 9 = 572,400 + 9 = 2,409
System prompt 300 tokens, query 100, answer 200, few-shot example 250, chunk 300; prices as in the figure; one training run a month for the fine-tune, one a quarter for fine-tune + RAG

The fine-tune saves 0.0015−0.0010=0.00050.0015-0.0010=0.0005 dollars per request, so N∗=27/0.0005=54,000N^\ast=27/0.0005=54{,}000 requests a month. With weekly knowledge retrains (U=4U=4) it is 108/0.0005=216,000108/0.0005=216{,}000. With 4 shots the prompt has Tin=1,400T_\text{in}=1{,}400 and costs exactly 0.001000.00100, the same as the fine-tune, so there is no saving to repay training. Caching a long static prefix (see Prompt caching) lowers cpromptc_\text{prompt} and moves N∗N^\ast further out.

python
1from dataclasses import dataclass
2
3@dataclass
4class Option:
5 name: str
6 tokens_in: int # prompt tokens per request
7 tokens_out: int = 200
8 price_mult: float = 1.0 # fine-tuned endpoints are often priced higher
9 per_query: float = 0.0 # retrieval: embedding + vector search
10 fixed: float = 0.0 # training runs per month
11
12 def monthly(self, n, p_in=0.5e-6, p_out=1.5e-6):
13 per_req = self.price_mult * (self.tokens_in * p_in + self.tokens_out * p_out) + self.per_query
14 return n * per_req + self.fixed
15
16train_run = 5_000 * 600 * 3 * 3e-6 # 9M training tokens at $3/M = $27
17options = [
18 Option("8-shot prompt", 300 + 100 + 8 * 250),
19 Option("RAG, k=4", 300 + 100 + 4 * 300, per_query=2e-4),
20 Option("fine-tune", 300 + 100, price_mult=2, fixed=train_run),
21]
22for n in (20_000, 1_000_000):
23 print(n, {o.name: round(o.monthly(n), 2) for o in options})
24# 20000 {'8-shot prompt': 30.0, 'RAG, k=4': 26.0, 'fine-tune': 47.0}
25# 1000000 {'8-shot prompt': 1500.0, 'RAG, k=4': 1300.0, 'fine-tune': 1027.0}
The cost model as a script (runs as is). Replace the prices with your provider's and the token counts with measurements from real traffic.

Combining them

A practical order of work

  1. 1Build an evaluation set from real queries, with expected answers or a rubric (see LLM-as-a-judge and RAG evaluation). Every later decision is measured on it.
  2. 2Write the best prompt: instructions, schema, a few examples. Measure. Many projects stop here.
  3. 3If answers lack facts or cite nothing, add retrieval (see RAG architecture). Measure retrieval recall separately from answer quality.
  4. 4If format, tone or a narrow skill still fails, fine-tune with LoRA on curated demonstrations, keeping the same prompt and retrieved context in the training examples that the model will see in production.
  5. 5Re-run the full eval after each change. Fine-tuning can quietly hurt general ability (forgetting) and the model's use of retrieved text.

Check yourself

A legal assistant must answer from a firm's 20,000 internal memos, which change daily, and must cite the memo used. The base model already writes in an acceptable style. What should you add to the prompt?

Check yourself

Using the worked-example prices, a 12-shot prompt is replaced by a fine-tune retrained once a month. What is the break-even volume?

Remember

  • ✓Knowledge (fresh, private, citable facts) goes in context via retrieval; behaviour (format, style, narrow skill) goes in the prompt, or in weights when the prompt cannot hold it.
  • ✓RAG beats fine-tuning for injecting facts (Ovadia et al. 2023), and fine-tuning on unknown facts raises hallucination (Gekhman et al. 2024).
  • ✓Cost: C(N)=N[m(Tinpin+Toutpout)+r]+UCtrainC(N)=N[m(T_\text{in}p_\text{in}+T_\text{out}p_\text{out})+r]+UC_\text{train}; break-even N∗=UCtrain/(cprompt−cft)N^\ast=UC_\text{train}/(c_\text{prompt}-c_\text{ft}).
  • ✓A fine-tune that removes too few prompt tokens to beat its price multiplier never pays back; frequent retrains push break-even out.
  • ✓Combine them: prompt + RAG + a LoRA trained on retrieved context with distractors (RAFT), measured on one eval set throughout.
← back to Generative AI roadmap