Definition
Retrieval-augmented generation answers an input by first retrieving passages from an external corpus and then generating the output conditioned on and . The name comes from Lewis et al. ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020): a DPR retriever over 21M Wikipedia passages and a BART-large generator, trained jointly by marginalising over the top- passages. In practice RAG now means the whole pipeline around an LLM: index, retrieve, rerank, prompt, generate with citations.
The input is a question or instruction; the output is text that should be correct and supported by the retrieved sources. Retrieval addresses three limits of a closed-book model: facts after its training cutoff, private documents it never saw, and the lack of a source to check.
Retrieve-then-read is older than LLMs. DrQA (Chen et al., 2017) retrieved Wikipedia articles with TF-IDF and extracted an answer span with an RNN reader (see Question answering). RAG replaced the extractive reader with a generator, and the sparse retriever with a learned dense one. The model-side variants, REALM, RAG-Sequence vs RAG-Token, FiD and RETRO, are covered in Retrieval-augmented transformers; this article is about the system.
The objective, and what the pipeline approximates
- DPR's question and passage encoders (two BERT-base models); passages are embedded once, offline
- the sum over the whole corpus is truncated to the best passages by maximum inner product search ( or in the paper)
- the generator; in an LLM pipeline the passages are simply concatenated into the prompt instead of marginalised
Concatenating the chunks into one prompt lets the LLM combine facts from several passages, at the cost of a longer context. Either way, the generator can only use what retrieval returns, so the retriever's recall@k caps the whole system.
Chunking
Documents are split before indexing. Lewis et al. used disjoint 100-word chunks of a December 2018 Wikipedia dump, 21M passages. Small chunks embed one idea each and retrieve precisely, but may cut an answer from its context; large chunks keep context but dilute the embedding and spend prompt tokens. Common practice: a few hundred tokens, some overlap between neighbours, split on headings or paragraphs, and prepend the document title to every chunk.
Sparse, dense and hybrid retrieval
- count of term in chunk ; the fraction saturates, so the 5th repeat adds little
- saturation and length normalisation; common defaults ,
- chunks containing , total chunks; this is Lucene's always-positive idf
BM25 matches exact terms: product codes, error numbers, names, rare words. It fails on paraphrase ("restore factory settings" vs "reset the router"). A dense bi-encoder (see Sentence embeddings, Contrastive language learning) matches meaning but blurs exact identifiers, since "E42" and "E17" look alike to it. Hybrid retrieval runs both and fuses the lists. Score scales differ (BM25 is unbounded, cosine is in ), so the robust choice is to fuse by rank:
- 1-based position of d in ranker r's list; a document a ranker did not return contributes 0
- the constant from Cormack, Clarke and Büttcher (SIGIR 2009); it damps the advantage of rank 1 over rank 2
A reranker then rescores the fused top 50–100 with a cross-encoder that reads query and chunk together (monoBERT, Nogueira and Cho, 2019). It is too slow to run on the whole corpus but much more accurate than either first-stage retriever.
Reference numbers
- DPR vs BM25
- Karpukhin et al. (EMNLP 2020), Natural Questions: top-20 retrieval accuracy 78.4% for DPR vs 59.1% for BM25; trained with in-batch negatives plus one BM25 hard negative
- RAG (2020)
- DPR + BART-large, 21M 100-word passages, or ; NQ exact match 44.5 (RAG-Sequence) vs 34.5 for closed-book T5-11B
- Retrieval metrics
- recall@k (share of gold chunks in the top ), MRR (mean of of the first gold chunk), nDCG@k (graded, position-discounted)
- Generation metrics
- faithfulness or groundedness (is every claim supported by the retrieved context?), answer relevance, citation precision; RAGAS (Es et al., 2023) automates these with an LLM judge
Worked example: BM25 and RRF on the mixed query
Query "internet offline since the firmware update" against the nine chunks in the figure (, avgdl terms after stop-word removal, , ). Gold chunks: d2 (firmware failed) and d4 (connection dropping).
Scoring d2 (11 terms) with BM25
- 1"firmware" occurs in 2 chunks: . It occurs twice in d2: saturation factor . Contribution .
- 2"update" occurs in 1 chunk (no stemming, so "updates" does not count): ; factor ; contribution .
- 3"internet", "offline", "since" do not occur in d2. Total . d4 gets (from "internet") and d6 (from "firmware"); no other chunk shares a term.
| Chunk | BM25 rank | Dense rank | RRF | Fused rank |
|---|---|---|---|---|
| d4 (gold) | 2 | 1 | 1/62 + 1/61 = 0.032522 | 1 |
| d2 (gold) | 1 | 4 | 1/61 + 1/64 = 0.032018 | 2 |
| d6 | 3 | 3 | 1/63 + 1/63 = 0.031746 | 3 |
| d9 | – | 2 | 1/62 = 0.016129 | 4 |
Top-2 recall: BM25 gives 1.0; dense gives 0.5; RRF gives 1.0. A chunk that only one ranker returns (d9) gets about half the score of a chunk both rankers return. RRF therefore rewards agreement. That is usually right, but a chunk ranked 3rd by both (d6) beats one ranked 1st by only one.
1from rank_bm25 import BM25Okapi2from sentence_transformers import SentenceTransformer, CrossEncoder3import numpy as np45chunks = [...] # list[str], ~100-300 words each6bm25 = BM25Okapi([c.lower().split() for c in chunks])7enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")8E = enc.encode(chunks, normalize_embeddings=True) # offline, (n, 384)9rerank = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")1011def retrieve(q, n=50, k=5):12 sparse = np.argsort(-bm25.get_scores(q.lower().split()))[:n]13 dense = np.argsort(-(E @ enc.encode(q, normalize_embeddings=True)))[:n]14 rrf = {}15 for ranking in (sparse, dense):16 for pos, d in enumerate(ranking):17 rrf[d] = rrf.get(d, 0.0) + 1 / (60 + pos + 1)18 cand = sorted(rrf, key=rrf.get, reverse=True)[:n]19 scores = rerank.predict([(q, chunks[d]) for d in cand])20 return [cand[i] for i in np.argsort(-scores)[:k]]2122q = "internet offline since the firmware update"23ctx = "\n".join(f"[{i+1}] {chunks[d]}" for i, d in enumerate(retrieve(q)))24prompt = f"Answer only from the sources and cite them as [n].\n{ctx}\n\nQ: {q}\nA:"Use when
- +Up-to-date and private knowledge without retraining; delete a document and it is gone
- +Answers can cite sources, so users and evaluators can check them
- +A small model with good retrieval can beat a much larger closed-book model on knowledge tasks
Avoid when
- −Recall@k caps quality: a missed chunk cannot be recovered by the generator
- −Extra latency and cost: retrieval, reranking and a longer prompt on every query
- −Retrieved text can be wrong, outdated or adversarial (prompt injection), and the model may still ignore it
Check yourself
Chunk A is ranked 1st by BM25 and not returned by the dense retriever. Chunk B is ranked 2nd by both. Which wins under RRF with k = 60?
Check yourself
Users search for error codes like "E42" and for vague descriptions like "internet keeps cutting out". Which retrieval setup fits best?
Remember
- ✓RAG: in the 2020 paper; in LLM pipelines, top- chunks concatenated into the prompt.
- ✓Pipeline: chunk → index (dense ANN + BM25 inverted index) → retrieve → fuse → rerank → prompt with citations → generate.
- ✓BM25 wins on exact terms, dense on paraphrase; reciprocal rank fusion combines them without calibrating scores.
- ✓DPR: 78.4% vs 59.1% top-20 accuracy over BM25 on Natural Questions.
- ✓Evaluate retrieval (recall@k, MRR) and generation (faithfulness, citation accuracy) separately; low recall caps everything.