# Retrieval-augmented generation (RAG)

> Retrieve relevant chunks from an index at query time and make the LLM answer from them: chunk, retrieve with BM25 and dense vectors, fuse, rerank, prompt with citations, and evaluate recall@k and faithfulness separately.

Source: https://www.mlfoundry.in/learn/retrieval-augmented-generation-rag  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**Retrieval-augmented generation:** answers an input $x$ by first retrieving passages $z$ from an external corpus and then generating the output conditioned on $x$ and $z$. The name comes from Lewis et al. ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020): a DPR retriever over 21M Wikipedia passages and a BART-large generator, trained jointly by marginalising over the top-$k$ passages. In practice RAG now means the whole pipeline around an LLM: index, retrieve, rerank, prompt, generate with citations.

The input is a question or instruction; the output is text that should be correct *and* supported by the retrieved sources. Retrieval addresses three limits of a closed-book model: facts after its training cutoff, private documents it never saw, and the lack of a source to check.

Retrieve-then-read is older than LLMs. DrQA (Chen et al., 2017) retrieved Wikipedia articles with TF-IDF and extracted an answer span with an RNN reader (see **Question answering**). RAG replaced the extractive reader with a generator, and the sparse retriever with a learned dense one. The model-side variants, REALM, RAG-Sequence vs RAG-Token, FiD and RETRO, are covered in **Retrieval-augmented transformers**; this article is about the system.

## The objective, and what the pipeline approximates

$$
p(y\mid x)\;\approx\;\sum_{z\,\in\,\text{top-}k}\;p_\eta(z\mid x)\;p_\theta(y\mid x,z),\qquad p_\eta(z\mid x)\propto \exp\!\big(E_d(z)^\top E_q(x)\big)
$$

- $E_q,\,E_d$: DPR's question and passage encoders (two BERT-base models); passages are embedded once, offline
- $\text{top-}k$: the sum over the whole corpus is truncated to the $k$ best passages by maximum inner product search ($k=5$ or $10$ in the paper)
- $p_\theta$: the generator; in an LLM pipeline the passages are simply concatenated into the prompt instead of marginalised

Concatenating the chunks into one prompt lets the LLM combine facts from several passages, at the cost of a longer context. Either way, the generator can only use what retrieval returns, so the retriever's **recall@k** caps the whole system.

> Figure: A production RAG pipeline. Offline, documents are chunked and indexed twice: dense vectors in an approximate-nearest-neighbour index and terms in an inverted index. Online, each retriever returns 50 candidates, reciprocal rank fusion merges them, a cross-encoder reranks, and the top 5 chunks go into the prompt with [n] markers so the answer can cite them. Retrieval and generation are evaluated separately.

## Chunking

Documents are split before indexing. Lewis et al. used disjoint 100-word chunks of a December 2018 Wikipedia dump, 21M passages. Small chunks embed one idea each and retrieve precisely, but may cut an answer from its context; large chunks keep context but dilute the embedding and spend prompt tokens. Common practice: a few hundred tokens, some overlap between neighbours, split on headings or paragraphs, and prepend the document title to every chunk.

## Sparse, dense and hybrid retrieval

$$
\text{BM25}(q,d)=\sum_{t\in q}\operatorname{idf}(t)\,\frac{f_{t,d}\,(k_1+1)}{f_{t,d}+k_1\big(1-b+b\,\frac{|d|}{\text{avgdl}}\big)},\qquad \operatorname{idf}(t)=\ln\!\Big(1+\frac{N-n_t+0.5}{n_t+0.5}\Big)
$$

- $f_{t,d}$: count of term $t$ in chunk $d$; the fraction saturates, so the 5th repeat adds little
- $k_1,\,b$: saturation and length normalisation; common defaults $k_1\in[1.2,2]$, $b=0.75$
- $n_t,\,N$: chunks containing $t$, total chunks; this is Lucene's always-positive idf

BM25 matches exact terms: product codes, error numbers, names, rare words. It fails on paraphrase ("restore factory settings" vs "reset the router"). A dense bi-encoder (see **Sentence embeddings**, **Contrastive language learning**) matches meaning but blurs exact identifiers, since "E42" and "E17" look alike to it. **Hybrid** retrieval runs both and fuses the lists. Score scales differ (BM25 is unbounded, cosine is in $[-1,1]$), so the robust choice is to fuse by rank:

$$
\operatorname{RRF}(d)=\sum_{r\in\text{rankers}}\frac{1}{k+\operatorname{rank}_r(d)},\qquad k=60
$$

- $\operatorname{rank}_r(d)$: 1-based position of d in ranker r's list; a document a ranker did not return contributes 0
- $k=60$: the constant from Cormack, Clarke and Büttcher (SIGIR 2009); it damps the advantage of rank 1 over rank 2

A **reranker** then rescores the fused top 50–100 with a cross-encoder that reads query and chunk together (monoBERT, Nogueira and Cho, 2019). It is too slow to run on the whole corpus but much more accurate than either first-stage retriever.

**Reference numbers**

- **DPR vs BM25:** Karpukhin et al. (EMNLP 2020), Natural Questions: top-20 retrieval accuracy 78.4% for DPR vs 59.1% for BM25; trained with in-batch negatives plus one BM25 hard negative
- **RAG (2020):** DPR + BART-large, 21M 100-word passages, $k=5$ or $10$; NQ exact match 44.5 (RAG-Sequence) vs 34.5 for closed-book T5-11B
- **Retrieval metrics:** recall@k (share of gold chunks in the top $k$), MRR (mean of $1/\text{rank}$ of the first gold chunk), nDCG@k (graded, position-discounted)
- **Generation metrics:** faithfulness or groundedness (is every claim supported by the retrieved context?), answer relevance, citation precision; RAGAS (Es et al., 2023) automates these with an LLM judge

> Figure: Real BM25, a toy dense encoder and RRF over nine help-centre chunks. Default (exact code query, hybrid, k = 2): d2 is first with RRF 0.0325 and recall@2 is 1.00. Switch to dense: the E17 chunk d9 (cosine 0.187) beats the gold d2 (0.160), so recall@1 is 0. Switch to the paraphrase query with BM25: only d5 shares a term, and recall stays 0.50 for every k. Averaged over the three queries at k = 2, BM25 and dense each score 0.833 and hybrid 1.000.

## Worked example: BM25 and RRF on the mixed query

Query "internet offline since the firmware update" against the nine chunks in the figure ($N=9$, avgdl $=7.78$ terms after stop-word removal, $k_1=1.2$, $b=0.75$). Gold chunks: d2 (firmware failed) and d4 (connection dropping).

**Scoring d2 (11 terms) with BM25**

1. "firmware" occurs in 2 chunks: $\operatorname{idf}=\ln(1+7.5/2.5)=1.3863$. It occurs twice in d2: saturation factor $2\cdot2.2/(2+1.2(0.25+0.75\cdot11/7.78))=1.2315$. Contribution $1.7072$.
2. "update" occurs in 1 chunk (no stemming, so "updates" does not count): $\operatorname{idf}=\ln(1+8.5/1.5)=1.8971$; factor $0.8551$; contribution $1.6222$.
3. "internet", "offline", "since" do not occur in d2. Total $\text{BM25}=3.3294$. d4 gets $1.9780$ (from "internet") and d6 $1.4454$ (from "firmware"); no other chunk shares a term.

Reciprocal rank fusion, k = 60 (computed in numpy; dense ranks from the figure's toy encoder)

| Chunk | BM25 rank | Dense rank | RRF | Fused rank |
| --- | --- | --- | --- | --- |
| d4 (gold) | 2 | 1 | 1/62 + 1/61 = 0.032522 | 1 |
| d2 (gold) | 1 | 4 | 1/61 + 1/64 = 0.032018 | 2 |
| d6 | 3 | 3 | 1/63 + 1/63 = 0.031746 | 3 |
| d9 | – | 2 | 1/62 = 0.016129 | 4 |

Top-2 recall: BM25 $\{d2,d4\}$ gives 1.0; dense $\{d4,d9\}$ gives 0.5; RRF $\{d4,d2\}$ gives 1.0. A chunk that only one ranker returns (d9) gets about half the score of a chunk both rankers return. RRF therefore rewards agreement. That is usually right, but a chunk ranked 3rd by both (d6) beats one ranked 1st by only one.

```python
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

chunks = [...]                                   # list[str], ~100-300 words each
bm25 = BM25Okapi([c.lower().split() for c in chunks])
enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
E = enc.encode(chunks, normalize_embeddings=True)            # offline, (n, 384)
rerank = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve(q, n=50, k=5):
    sparse = np.argsort(-bm25.get_scores(q.lower().split()))[:n]
    dense = np.argsort(-(E @ enc.encode(q, normalize_embeddings=True)))[:n]
    rrf = {}
    for ranking in (sparse, dense):
        for pos, d in enumerate(ranking):
            rrf[d] = rrf.get(d, 0.0) + 1 / (60 + pos + 1)
    cand = sorted(rrf, key=rrf.get, reverse=True)[:n]
    scores = rerank.predict([(q, chunks[d]) for d in cand])
    return [cand[i] for i in np.argsort(-scores)[:k]]

q = "internet offline since the firmware update"
ctx = "\n".join(f"[{i+1}] {chunks[d]}" for i, d in enumerate(retrieve(q)))
prompt = f"Answer only from the sources and cite them as [n].\n{ctx}\n\nQ: {q}\nA:"
```

Hybrid retrieval with RRF, cross-encoder reranking and a cited prompt (rank_bm25 + sentence-transformers)

> **Debug retrieval before the prompt:** When a RAG answer is wrong, first check whether the gold chunk was in the top $k$. If recall@k is low, no prompt or model change will fix it; fix chunking, add BM25, add a reranker or raise $k$. If the chunk *was* there and the answer still contradicts it, it is a faithfulness problem (see **Hallucination detection**). Order also matters: models use passages at the start or end of a long context better than those in the middle (see **Long-context NLP**).

> **Knowledge in the index, skills in the weights:** Updating a RAG system's knowledge means re-indexing documents, not retraining. Fine-tuning (see **Supervised fine-tuning (SFT)**) is better for changing behaviour and format. The two combine: fine-tune the model to cite and to refuse when the context is silent, and keep the facts in the index.

**Strengths**

- Up-to-date and private knowledge without retraining; delete a document and it is gone
- Answers can cite sources, so users and evaluators can check them
- A small model with good retrieval can beat a much larger closed-book model on knowledge tasks

**Limitations**

- Recall@k caps quality: a missed chunk cannot be recovered by the generator
- Extra latency and cost: retrieval, reranking and a longer prompt on every query
- Retrieved text can be wrong, outdated or adversarial (prompt injection), and the model may still ignore it

**Question:** Chunk A is ranked 1st by BM25 and not returned by the dense retriever. Chunk B is ranked 2nd by both. Which wins under RRF with k = 60?

- A. A, with 1/61 = 0.0164
- B. B, with 2/62 = 0.0323
- C. They tie
- D. It depends on the raw BM25 score

**Answer:** B. B, with 2/62 = 0.0323 — $\operatorname{RRF}(A)=1/61=0.0164$ and $\operatorname{RRF}(B)=1/62+1/62=0.0323$. RRF uses ranks only, so agreement between rankers counts for almost twice as much as one first place.

**Question:** Users search for error codes like "E42" and for vague descriptions like "internet keeps cutting out". Which retrieval setup fits best?

- A. BM25 only: it handles both
- B. Dense only: embeddings understand codes
- C. Hybrid BM25 + dense with rank fusion, then a reranker
- D. No retrieval: put the whole manual in the prompt

**Answer:** C. Hybrid BM25 + dense with rank fusion, then a reranker — BM25 matches the exact code; the dense encoder matches the paraphrase; fusion keeps both candidates and the cross-encoder orders them. In the figure, at k = 2 each single retriever averages recall 0.833 over three such queries and the hybrid reaches 1.000.

**Key points**

- RAG: $p(y\mid x)\approx\sum_{z\in\text{top-}k}p_\eta(z\mid x)\,p_\theta(y\mid x,z)$ in the 2020 paper; in LLM pipelines, top-$k$ chunks concatenated into the prompt.
- Pipeline: chunk → index (dense ANN + BM25 inverted index) → retrieve → fuse → rerank → prompt with citations → generate.
- BM25 wins on exact terms, dense on paraphrase; reciprocal rank fusion $\sum 1/(60+\text{rank})$ combines them without calibrating scores.
- DPR: 78.4% vs 59.1% top-20 accuracy over BM25 on Natural Questions.
- Evaluate retrieval (recall@k, MRR) and generation (faithfulness, citation accuracy) separately; low recall caps everything.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
