# Fine-tuning vs RAG vs prompting

> Prompt first; add RAG when the model lacks facts (fresh, private, citable), fine-tune when it lacks behaviour (format, style, a narrow skill). A fine-tune saves money only if its shorter prompts repay training.

Source: https://www.mlfoundry.in/learn/fine-tuning-vs-rag-vs-prompting  
Section: Generative AI › Customization  
Level: core · 8 min read · updated 2026-10-21

**Fine-tuning vs RAG vs prompting:** is the choice of where to put what a model needs for a task. **Prompting** puts instructions and examples in the context window at inference time (in-context learning, Brown et al., "Language Models are Few-Shot Learners", 2020). **Retrieval-augmented generation** fetches relevant documents per query and puts them in the context (Lewis et al., NeurIPS 2020). **Fine-tuning** changes the weights with gradient updates on task examples. All three can be combined in one system.

The question comes up as soon as a base model almost works. A support bot gives generic answers, invents a refund policy, or ignores the house tone. Each failure has a different fix. Pick the wrong one and you pay for it. Fine-tuning on the policy manual will not make the bot cite this week's policy. A 4,000-token prompt of style rules costs money on every request.

The useful split is **knowledge vs behaviour**. Knowledge is facts the model never saw or that change: last week's prices, a private wiki, a statute amended after the training cutoff. Behaviour is how the model answers: output schema, tone, length, domain jargon, a task procedure. Knowledge belongs in context, where it can be updated and cited. Behaviour belongs in the prompt or, if the prompt cannot hold it, in the weights.

> Figure: The decision flow. Start with the best prompt you can write and an evaluation set of 50–200 real cases. If it fails, classify the failure: missing knowledge leads to RAG, wrong behaviour to fine-tuning (usually LoRA), and both to a fine-tuned model that is trained to read retrieved documents (RAFT). The prompt stays in every option.

## Why knowledge goes in context

Ovadia et al. ("Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", 2023) compared the two on knowledge-heavy MMLU subjects and on questions about events after the training cutoff. RAG beat unsupervised fine-tuning on the same documents in nearly every setting, for both old and new facts. Fine-tuning only began to absorb new facts when each fact appeared in many paraphrases.

Gekhman et al. ("Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?", EMNLP 2024) found a second cost. Fine-tuning examples with facts the model did not already know were learned much more slowly than familiar ones. Once they were learned, the model hallucinated more on other questions (see **Hallucination**). Zhou et al.'s LIMA paper (2023) states the converse as the *superficial alignment hypothesis*: almost all knowledge comes from pretraining, and fine-tuning mostly teaches format and style.

Retrieval has practical advantages too. Updating the index takes minutes, not a training run. Answers can cite their source passage. Access control can filter documents per user. Deleting a document removes it from future answers, while removing a fact from weights has no reliable method.

## Why behaviour can go in weights

A prompt can carry a schema, a persona and a few examples (see **Few-shot prompting**, **System prompts and role prompting** and **Structured output (JSON mode)**). It runs out in three ways. The behaviour needs more examples than fit or than you want to pay for. Compliance is inconsistent: the model follows the style in 90% of answers, and the rest break a parser. Or the task needs a skill the base model only shows after training, such as a new output language, a domain classifier or a tool-call format. Fine-tuning on 500–5,000 good demonstrations (see **Supervised fine-tuning (SFT)** and **Dataset curation for fine-tuning**) puts the behaviour in the weights. It also lets a smaller model match a larger prompted one on a narrow task (see **Knowledge distillation**).

## Cost: fixed vs per-request

$$
C(N)=N\big[m\,(T_\text{in}\,p_\text{in}+T_\text{out}\,p_\text{out})+r\big]+U\,C_\text{train}
$$

- $N$: requests per month
- $T_\text{in},T_\text{out}$: prompt and answer tokens per request; few-shot examples and retrieved chunks add to $T_\text{in}$
- $p_\text{in},p_\text{out}$: price per token of the base model
- $m$: price multiplier of the serving model; hosted fine-tuned endpoints are often priced above the base model
- $r$: retrieval cost per query (embedding + vector search); 0 without RAG
- $U\,C_\text{train}$: training runs per month times cost per run; 0 without fine-tuning

$$
N^\ast=\frac{U\,C_\text{train}}{c_\text{prompt}-c_\text{ft}},\qquad c=m\,(T_\text{in}\,p_\text{in}+T_\text{out}\,p_\text{out})+r
$$

- $N^\ast$: break-even volume above which the fine-tune is cheaper than prompting
- $c_\text{prompt}-c_\text{ft}$: per-request saving; if it is $\le 0$ the fine-tune never pays back

Fine-tuning swaps a per-request cost (long prompts) for a fixed cost (training runs). It pays off only at volume, and only if it removes enough tokens to beat the price multiplier. Every knowledge update that forces a retrain adds to $U$. That is the economic case for keeping changing facts in an index. The engineering cost of building a dataset usually dwarfs the compute; the equation leaves it out.

The three options compared

| Aspect | Prompting | RAG | Fine-tuning |
| --- | --- | --- | --- |
| Changes | context (instructions, examples) | context (retrieved passages) | weights |
| Best for | anything the base model can already do | facts: fresh, private, long-tail, citable | behaviour: format, style, narrow skill |
| Update | edit the prompt, seconds | re-index, minutes | new data + training run, hours |
| Data needed | a few examples + eval set | a document corpus | hundreds to thousands of curated demonstrations |
| Per-request cost | grows with prompt length | +k chunks + retrieval call | short prompt, but m × price or own hosting |
| Main risk | prompt drift, injection | bad retrieval, context overload | stale facts, forgetting, more hallucination on new facts |

**Results worth knowing**

- **Knowledge injection:** Ovadia et al. (2023): RAG outperformed unsupervised fine-tuning on MMLU subjects and on post-cutoff current-events questions; combining them gave no consistent gain over RAG alone.
- **New facts via SFT:** Gekhman et al. (2024): unknown-fact examples are fitted late in training and raise hallucination rates; early stopping limits the damage.
- **Long context:** Liu et al., "Lost in the Middle" (TACL 2024): accuracy is highest when the relevant passage sits at the start or end of a long context and drops for the middle, so stuffing everything into the prompt is not a free substitute for retrieval.
- **RAFT:** Zhang et al. (2024): fine-tune on (question, retrieved documents including distractors, chain-of-thought answer quoting the right document); it improved domain RAG on PubMed, HotpotQA and Gorilla API questions over RAG with the base model.

> Figure: Monthly cost of each option under the cost equation, with illustrative prices ($0.50/M input, $1.50/M output, fine-tuned model at 2×, $27 per training run). Defaults: 20,000 requests, 8 shots, k = 4, one retrain a month, no requirement. Prompting costs $30 and the fine-tune $47; break-even is 54,000 requests. At 1M requests the fine-tune costs $1,027 against $1,500. Four updates a month push break-even to 216,000. At 4 shots or fewer the fine-tune never pays back. Toggle 'facts the model lacks' and RAG wins at $26; toggle both and only fine-tune + RAG remains ($57).

## Worked example: break-even

System prompt 300 tokens, query 100, answer 200, few-shot example 250, chunk 300; prices as in the figure; one training run a month for the fine-tune, one a quarter for fine-tune + RAG

| Option | T_in | $ / request | 20k / month ($) | 1M / month ($) |
| --- | --- | --- | --- | --- |
| 8-shot prompt | 2,400 | 1,200 + 300 µ$ = 0.00150 | 30 | 1,500 |
| RAG, k = 4 | 1,600 | 800 + 300 + 200 µ$ = 0.00130 | 26 | 1,300 |
| Fine-tune | 400 | 2 × (200 + 300) µ$ = 0.00100 | 20 + 27 = 47 | 1,000 + 27 = 1,027 |
| Fine-tune + RAG | 1,600 | 2 × 1,100 + 200 µ$ = 0.00240 | 48 + 9 = 57 | 2,400 + 9 = 2,409 |

The fine-tune saves $0.0015-0.0010=0.0005$ dollars per request, so $N^\ast=27/0.0005=54{,}000$ requests a month. With weekly knowledge retrains ($U=4$) it is $108/0.0005=216{,}000$. With 4 shots the prompt has $T_\text{in}=1{,}400$ and costs exactly $0.00100$, the same as the fine-tune, so there is no saving to repay training. Caching a long static prefix (see **Prompt caching**) lowers $c_\text{prompt}$ and moves $N^\ast$ further out.

```python
from dataclasses import dataclass

@dataclass
class Option:
    name: str
    tokens_in: int          # prompt tokens per request
    tokens_out: int = 200
    price_mult: float = 1.0 # fine-tuned endpoints are often priced higher
    per_query: float = 0.0  # retrieval: embedding + vector search
    fixed: float = 0.0      # training runs per month

    def monthly(self, n, p_in=0.5e-6, p_out=1.5e-6):
        per_req = self.price_mult * (self.tokens_in * p_in + self.tokens_out * p_out) + self.per_query
        return n * per_req + self.fixed

train_run = 5_000 * 600 * 3 * 3e-6               # 9M training tokens at $3/M = $27
options = [
    Option("8-shot prompt", 300 + 100 + 8 * 250),
    Option("RAG, k=4", 300 + 100 + 4 * 300, per_query=2e-4),
    Option("fine-tune", 300 + 100, price_mult=2, fixed=train_run),
]
for n in (20_000, 1_000_000):
    print(n, {o.name: round(o.monthly(n), 2) for o in options})
# 20000 {'8-shot prompt': 30.0, 'RAG, k=4': 26.0, 'fine-tune': 47.0}
# 1000000 {'8-shot prompt': 1500.0, 'RAG, k=4': 1300.0, 'fine-tune': 1027.0}
```

The cost model as a script (runs as is). Replace the prices with your provider's and the token counts with measurements from real traffic.

## Combining them

**A practical order of work**

1. Build an evaluation set from real queries, with expected answers or a rubric (see **LLM-as-a-judge** and **RAG evaluation**). Every later decision is measured on it.
2. Write the best prompt: instructions, schema, a few examples. Measure. Many projects stop here.
3. If answers lack facts or cite nothing, add retrieval (see **RAG architecture**). Measure retrieval recall separately from answer quality.
4. If format, tone or a narrow skill still fails, fine-tune with **LoRA** on curated demonstrations, keeping the same prompt and retrieved context in the training examples that the model will see in production.
5. Re-run the full eval after each change. Fine-tuning can quietly hurt general ability (forgetting) and the model's use of retrieved text.

> **Fine-tuning to teach facts:** Training on a product manual so that the model "knows" it is the most common mistake. The model learns the manual's style, recalls facts unreliably, cannot cite them, and keeps outdated ones after the manual changes. Put the manual in an index. Fine-tune only on the *behaviour* of answering from retrieved passages.

> **Train the way you serve:** If production prompts contain retrieved chunks, training examples must too, including irrelevant ones. RAFT trains with distractor documents and leaves the relevant document out of some examples, so the model learns to quote the right passage and ignore the rest. A model fine-tuned on bare question–answer pairs may ignore context it was never trained to read.

**Question:** A legal assistant must answer from a firm's 20,000 internal memos, which change daily, and must cite the memo used. The base model already writes in an acceptable style. What should you add to the prompt?

- A. Fine-tune on all memos
- B. Retrieval over the memos
- C. More few-shot examples
- D. Fine-tune and drop the prompt

**Answer:** B. Retrieval over the memos — The gap is knowledge that is private, changes daily and must be cited, which is exactly what retrieval provides. Fine-tuning on the memos would go stale within a day, could not cite, and per Ovadia et al. absorbs facts worse than RAG. Style is already fine, so no fine-tune is needed.

**Question:** Using the worked-example prices, a 12-shot prompt is replaced by a fine-tune retrained once a month. What is the break-even volume?

- A. 18,000 requests
- B. 27,000 requests
- C. 54,000 requests
- D. Never

**Answer:** B. 27,000 requests — 12 shots give $T_\text{in}=300+100+3{,}000=3{,}400$, so $c_\text{prompt}=0.0017+0.0003=0.0020$ dollars. The fine-tune costs $0.0010$, saving $0.0010$ per request. $N^\ast=27/0.0010=27{,}000$.

**Key points**

- Knowledge (fresh, private, citable facts) goes in context via retrieval; behaviour (format, style, narrow skill) goes in the prompt, or in weights when the prompt cannot hold it.
- RAG beats fine-tuning for injecting facts (Ovadia et al. 2023), and fine-tuning on unknown facts raises hallucination (Gekhman et al. 2024).
- Cost: $C(N)=N[m(T_\text{in}p_\text{in}+T_\text{out}p_\text{out})+r]+UC_\text{train}$; break-even $N^\ast=UC_\text{train}/(c_\text{prompt}-c_\text{ft})$.
- A fine-tune that removes too few prompt tokens to beat its price multiplier never pays back; frequent retrains push break-even out.
- Combine them: prompt + RAG + a LoRA trained on retrieved context with distractors (RAFT), measured on one eval set throughout.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
