Retrieval-augmented generation (RAG)

Retrieve relevant chunks from an index at query time and make the LLM answer from them: chunk, retrieve with BM25 and dense vectors, fuse, rerank, prompt with citations, and evaluate recall@k and faithfulness separately.

8 min read767 wordscore0 viewsupdated 2026-10-15

Definition

Retrieval-augmented generation answers an input xx by first retrieving passages zz from an external corpus and then generating the output conditioned on xx and zz. The name comes from Lewis et al. ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020): a DPR retriever over 21M Wikipedia passages and a BART-large generator, trained jointly by marginalising over the top-kk passages. In practice RAG now means the whole pipeline around an LLM: index, retrieve, rerank, prompt, generate with citations.

The input is a question or instruction; the output is text that should be correct and supported by the retrieved sources. Retrieval addresses three limits of a closed-book model: facts after its training cutoff, private documents it never saw, and the lack of a source to check.

Retrieve-then-read is older than LLMs. DrQA (Chen et al., 2017) retrieved Wikipedia articles with TF-IDF and extracted an answer span with an RNN reader (see Question answering). RAG replaced the extractive reader with a generator, and the sparse retriever with a learned dense one. The model-side variants, REALM, RAG-Sequence vs RAG-Token, FiD and RETRO, are covered in Retrieval-augmented transformers; this article is about the system.

The objective, and what the pipeline approximates

p(y∣x)  ≈  ∑z ∈ top-k  pη(z∣x)  pθ(y∣x,z),pη(z∣x)∝exp⁡ ⁣(Ed(z)⊤Eq(x))p(y\mid x)\;\approx\;\sum_{z\,\in\,\text{top-}k}\;p_\eta(z\mid x)\;p_\theta(y\mid x,z),\qquad p_\eta(z\mid x)\propto \exp\!\big(E_d(z)^\top E_q(x)\big)
Eq, EdE_q,\,E_d
DPR's question and passage encoders (two BERT-base models); passages are embedded once, offline
top-k\text{top-}k
the sum over the whole corpus is truncated to the kk best passages by maximum inner product search (k=5k=5 or 1010 in the paper)
pθp_\theta
the generator; in an LLM pipeline the passages are simply concatenated into the prompt instead of marginalised

Concatenating the chunks into one prompt lets the LLM combine facts from several passages, at the cost of a longer context. Either way, the generator can only use what retrieval returns, so the retriever's recall@k caps the whole system.

offline · indexing (once, and again when documents change)help-centrepages(9 in thefigure below)chunk~100–300 wordswith overlap,keep the titleembed E_d(chunk)tokenisevector index (ANN)inverted indexonline · per query (BM25 reads the inverted index, dense the ANN index with E_q(query))“internet offline sincethe firmware update”BM25 top-50dense top-50RRF fuseΣ 1/(60+r)→ top-50cross-encoderrerank (q, chunk)→ top-5Answer only from the sources; cite [n].[1] Error code E42 means the firmware update…[2] If the internet connection keeps dropping…Q: internet offline since the firmware updateLLMgeneratorReinstall thefirmware from theadmin page, thenrestart the modem [1][2]retrieval: recall@k, MRR, nDCGon labelled (query, gold chunk) pairsgeneration: faithfulness (claimssupported by [1]…[k]), answerrelevance, citation accuracy
fig.A production RAG pipeline. Offline, documents are chunked and indexed twice: dense vectors in an approximate-nearest-neighbour index and terms in an inverted index. Online, each retriever returns 50 candidates, reciprocal rank fusion merges them, a cross-encoder reranks, and the top 5 chunks go into the prompt with [n] markers so the answer can cite them. Retrieval and generation are evaluated separately.

Chunking

Documents are split before indexing. Lewis et al. used disjoint 100-word chunks of a December 2018 Wikipedia dump, 21M passages. Small chunks embed one idea each and retrieve precisely, but may cut an answer from its context; large chunks keep context but dilute the embedding and spend prompt tokens. Common practice: a few hundred tokens, some overlap between neighbours, split on headings or paragraphs, and prepend the document title to every chunk.

Sparse, dense and hybrid retrieval

BM25(q,d)=∑t∈qidf⁡(t) ft,d (k1+1)ft,d+k1(1−b+b ∣d∣avgdl),idf⁡(t)=ln⁡ ⁣(1+N−nt+0.5nt+0.5)\text{BM25}(q,d)=\sum_{t\in q}\operatorname{idf}(t)\,\frac{f_{t,d}\,(k_1+1)}{f_{t,d}+k_1\big(1-b+b\,\frac{|d|}{\text{avgdl}}\big)},\qquad \operatorname{idf}(t)=\ln\!\Big(1+\frac{N-n_t+0.5}{n_t+0.5}\Big)
ft,df_{t,d}
count of term tt in chunk dd; the fraction saturates, so the 5th repeat adds little
k1, bk_1,\,b
saturation and length normalisation; common defaults k1∈[1.2,2]k_1\in[1.2,2], b=0.75b=0.75
nt, Nn_t,\,N
chunks containing tt, total chunks; this is Lucene's always-positive idf

BM25 matches exact terms: product codes, error numbers, names, rare words. It fails on paraphrase ("restore factory settings" vs "reset the router"). A dense bi-encoder (see Sentence embeddings, Contrastive language learning) matches meaning but blurs exact identifiers, since "E42" and "E17" look alike to it. Hybrid retrieval runs both and fuses the lists. Score scales differ (BM25 is unbounded, cosine is in [−1,1][-1,1]), so the robust choice is to fuse by rank:

RRF⁡(d)=∑r∈rankers1k+rank⁡r(d),k=60\operatorname{RRF}(d)=\sum_{r\in\text{rankers}}\frac{1}{k+\operatorname{rank}_r(d)},\qquad k=60
rank⁡r(d)\operatorname{rank}_r(d)
1-based position of d in ranker r's list; a document a ranker did not return contributes 0
k=60k=60
the constant from Cormack, Clarke and Büttcher (SIGIR 2009); it damps the advantage of rank 1 over rank 2

A reranker then rescores the fused top 50–100 with a cross-encoder that reads query and chunk together (monoBERT, Nogueira and Cho, 2019). It is too slow to run on the whole corpus but much more accurate than either first-stage retriever.

Reference numbers

DPR vs BM25
Karpukhin et al. (EMNLP 2020), Natural Questions: top-20 retrieval accuracy 78.4% for DPR vs 59.1% for BM25; trained with in-batch negatives plus one BM25 hard negative
RAG (2020)
DPR + BART-large, 21M 100-word passages, k=5k=5 or 1010; NQ exact match 44.5 (RAG-Sequence) vs 34.5 for closed-book T5-11B
Retrieval metrics
recall@k (share of gold chunks in the top kk), MRR (mean of 1/rank1/\text{rank} of the first gold chunk), nDCG@k (graded, position-discounted)
Generation metrics
faithfulness or groundedness (is every claim supported by the retrieved context?), answer relevance, citation precision; RAGAS (Es et al., 2023) automates these with an LLM judge
query: “what does E42 mean”BM25 terms: e42, mean · ★ = gold chunk (1)hybrid (RRF) ranking (top-2 go into the prompt)Σ 1/(60+rank)bm25#dense#1★ d2 Error code E42 means the firmware update failed; reinst…0.0325122d9 Error code E17 indicates a weak signal; move the router c…0.0164–13d7 The guest network isolates visitors from your devices.0.0159–34d8 To change the Wi-Fi password, open the admin page and sel…0.0156–45d4 If the internet connection keeps dropping, restart the mo…0.0154–56d1 To reset the router, hold the reset button for ten second…0.0152–67d6 Firmware updates are released monthly and install automat…0.0149–78d5 Restoring factory defaults erases all settings, including…0.0147–89d3 The router supports WPA3 encryption for wireless networks.0.0145–9recall@2 = 1.00 · first gold at rank 1 (RR 1.000) · prompt: 2 chunks, 29 wordsmean recall@2 over the 3 queries:BM250.833dense0.833hybrid (RRF)1.000BM25 needs shared terms;dense blurs exact codes;RRF only needs ranks.
query
retriever
fig.Real BM25, a toy dense encoder and RRF over nine help-centre chunks. Default (exact code query, hybrid, k = 2): d2 is first with RRF 0.0325 and recall@2 is 1.00. Switch to dense: the E17 chunk d9 (cosine 0.187) beats the gold d2 (0.160), so recall@1 is 0. Switch to the paraphrase query with BM25: only d5 shares a term, and recall stays 0.50 for every k. Averaged over the three queries at k = 2, BM25 and dense each score 0.833 and hybrid 1.000.

Worked example: BM25 and RRF on the mixed query

Query "internet offline since the firmware update" against the nine chunks in the figure (N=9N=9, avgdl =7.78=7.78 terms after stop-word removal, k1=1.2k_1=1.2, b=0.75b=0.75). Gold chunks: d2 (firmware failed) and d4 (connection dropping).

Scoring d2 (11 terms) with BM25

  1. 1"firmware" occurs in 2 chunks: idf⁡=ln⁡(1+7.5/2.5)=1.3863\operatorname{idf}=\ln(1+7.5/2.5)=1.3863. It occurs twice in d2: saturation factor 2⋅2.2/(2+1.2(0.25+0.75⋅11/7.78))=1.23152\cdot2.2/(2+1.2(0.25+0.75\cdot11/7.78))=1.2315. Contribution 1.70721.7072.
  2. 2"update" occurs in 1 chunk (no stemming, so "updates" does not count): idf⁡=ln⁡(1+8.5/1.5)=1.8971\operatorname{idf}=\ln(1+8.5/1.5)=1.8971; factor 0.85510.8551; contribution 1.62221.6222.
  3. 3"internet", "offline", "since" do not occur in d2. Total BM25=3.3294\text{BM25}=3.3294. d4 gets 1.97801.9780 (from "internet") and d6 1.44541.4454 (from "firmware"); no other chunk shares a term.
ChunkBM25 rankDense rankRRFFused rank
d4 (gold)211/62 + 1/61 = 0.0325221
d2 (gold)141/61 + 1/64 = 0.0320182
d6331/63 + 1/63 = 0.0317463
d9–21/62 = 0.0161294
Reciprocal rank fusion, k = 60 (computed in numpy; dense ranks from the figure's toy encoder)

Top-2 recall: BM25 {d2,d4}\{d2,d4\} gives 1.0; dense {d4,d9}\{d4,d9\} gives 0.5; RRF {d4,d2}\{d4,d2\} gives 1.0. A chunk that only one ranker returns (d9) gets about half the score of a chunk both rankers return. RRF therefore rewards agreement. That is usually right, but a chunk ranked 3rd by both (d6) beats one ranked 1st by only one.

python
1from rank_bm25 import BM25Okapi
2from sentence_transformers import SentenceTransformer, CrossEncoder
3import numpy as np
4
5chunks = [...] # list[str], ~100-300 words each
6bm25 = BM25Okapi([c.lower().split() for c in chunks])
7enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
8E = enc.encode(chunks, normalize_embeddings=True) # offline, (n, 384)
9rerank = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
10
11def retrieve(q, n=50, k=5):
12 sparse = np.argsort(-bm25.get_scores(q.lower().split()))[:n]
13 dense = np.argsort(-(E @ enc.encode(q, normalize_embeddings=True)))[:n]
14 rrf = {}
15 for ranking in (sparse, dense):
16 for pos, d in enumerate(ranking):
17 rrf[d] = rrf.get(d, 0.0) + 1 / (60 + pos + 1)
18 cand = sorted(rrf, key=rrf.get, reverse=True)[:n]
19 scores = rerank.predict([(q, chunks[d]) for d in cand])
20 return [cand[i] for i in np.argsort(-scores)[:k]]
21
22q = "internet offline since the firmware update"
23ctx = "\n".join(f"[{i+1}] {chunks[d]}" for i, d in enumerate(retrieve(q)))
24prompt = f"Answer only from the sources and cite them as [n].\n{ctx}\n\nQ: {q}\nA:"
Hybrid retrieval with RRF, cross-encoder reranking and a cited prompt (rank_bm25 + sentence-transformers)

Use when

  • +Up-to-date and private knowledge without retraining; delete a document and it is gone
  • +Answers can cite sources, so users and evaluators can check them
  • +A small model with good retrieval can beat a much larger closed-book model on knowledge tasks

Avoid when

  • −Recall@k caps quality: a missed chunk cannot be recovered by the generator
  • −Extra latency and cost: retrieval, reranking and a longer prompt on every query
  • −Retrieved text can be wrong, outdated or adversarial (prompt injection), and the model may still ignore it

Check yourself

Chunk A is ranked 1st by BM25 and not returned by the dense retriever. Chunk B is ranked 2nd by both. Which wins under RRF with k = 60?

Check yourself

Users search for error codes like "E42" and for vague descriptions like "internet keeps cutting out". Which retrieval setup fits best?

Remember

  • ✓RAG: p(y∣x)≈∑z∈top-kpη(z∣x) pθ(y∣x,z)p(y\mid x)\approx\sum_{z\in\text{top-}k}p_\eta(z\mid x)\,p_\theta(y\mid x,z) in the 2020 paper; in LLM pipelines, top-kk chunks concatenated into the prompt.
  • ✓Pipeline: chunk → index (dense ANN + BM25 inverted index) → retrieve → fuse → rerank → prompt with citations → generate.
  • ✓BM25 wins on exact terms, dense on paraphrase; reciprocal rank fusion ∑1/(60+rank)\sum 1/(60+\text{rank}) combines them without calibrating scores.
  • ✓DPR: 78.4% vs 59.1% top-20 accuracy over BM25 on Natural Questions.
  • ✓Evaluate retrieval (recall@k, MRR) and generation (faithfulness, citation accuracy) separately; low recall caps everything.
← back to Deep Learning roadmap