Retrieval-augmented transformers

A retrieval-augmented transformer looks up text or stored hidden states in an external datastore and conditions its prediction on them: kNN-LM mixes a neighbour distribution into the output, REALM and RAG marginalise over retrieved passages, RETRO cross-attends to retrieved chunks.

16 min read1047 wordscore0 viewsupdated 2026-10-12

Definition

Retrieval-augmented transformer is a Transformer language model that, for each input, retrieves items from a large external datastore (passages, chunks, or stored (context, next-token) pairs) by nearest-neighbour search over learned embeddings, and conditions its prediction on them. Knowledge then lives partly in an index that can be inspected, replaced or extended without retraining, rather than only in the weights. The main designs differ in where retrieval enters: at the output distribution (kNN-LM), as a latent variable that is marginalised (REALM, RAG), or through Cross-attention inside the network (Fusion-in-Decoder, RETRO).

Closed-book language models store facts in their parameters. Updating a fact means retraining, the model cannot point to a source, and rare facts need very large models. Open-domain QA systems had long used a retrieve-then-read pipeline (Chen et al.'s DrQA, 2017) with a fixed retriever such as TF-IDF. In 2020 several papers made retrieval part of the neural model. Khandelwal, Levy, Jurafsky, Zettlemoyer and Lewis (kNN-LM, ICLR 2020) interpolated a pretrained LM with nearest neighbours in its own hidden-state space. Guu, Lee, Tung, Pasupat and Chang (REALM, ICML 2020) trained a dense retriever jointly with a masked language model. Lewis et al. (RAG, NeurIPS 2020) combined a dense retriever with a seq2seq generator. Izacard and Grave's Fusion-in-Decoder (EACL 2021) and Borgeaud et al.'s RETRO (ICML 2022) moved retrieved text into cross-attention, RETRO with a datastore of 2 trillion tokens.

kNN-LM: retrieval at the output

pkNN(y∣x) ∝∑(ki,vi)∈Nk(x)1[y=vi] exp⁡ ⁣(−d(ki,f(x))),p(y∣x)=λ pkNN(y∣x)+(1−λ) pLM(y∣x)p_{\text{kNN}}(y\mid x)\ \propto \sum_{(k_i,v_i)\in\mathcal N_k(x)} \mathbb 1[y=v_i]\,\exp\!\big(-d(k_i,f(x))\big),\qquad p(y\mid x)=\lambda\,p_{\text{kNN}}(y\mid x)+(1-\lambda)\,p_{\text{LM}}(y\mid x)
f(x)f(x)
the LM's hidden representation of the context xx (in the paper the 1024-d input to the final feed-forward layer)
(ki,vi)(k_i,v_i)
datastore entry: key f(ci)f(c_i) for a training context cic_i, value the token wiw_i that followed it; one entry per training token
Nk(x)\mathcal N_k(x)
the kk keys nearest to f(x)f(x), found with an approximate index (FAISS)
dd
squared Euclidean distance (it worked better than the inner product)
λ\lambda
interpolation weight tuned on validation data (0.25 for WikiText-103)

No parameters are trained: the datastore is one forward pass over the training data. The neighbours help most on rare patterns such as names and factual phrases, which the softmax layer generalises poorly but which appear verbatim in similar contexts. Because the datastore is separate from the model, a model trained on one corpus can be given the datastore of another, in-domain one, and its perplexity there drops without fine-tuning.

REALM and RAG: retrieved passages as a latent variable

pη(z∣x)=exp⁡(Ed(z)⊤Eq(x))∑z′exp⁡(Ed(z′)⊤Eq(x)),p(y∣x)≈∑z∈top-kpη(z∣x) pθ(y∣x,z)p_\eta(z\mid x)=\frac{\exp\big(E_d(z)^\top E_q(x)\big)}{\sum_{z'}\exp\big(E_d(z')^\top E_q(x)\big)},\qquad p(y\mid x)\approx\sum_{z\in\text{top-}k}p_\eta(z\mid x)\,p_\theta(y\mid x,z)
Eq,EdE_q,E_d
query and document encoders (BERT-based bi-encoders); documents are embedded once and indexed for maximum inner product search (MIPS)
zz
a retrieved passage, treated as a latent variable: there is no label saying which passage is right
pθp_\theta
the reader: a masked LM in REALM, a BART seq2seq generator in RAG
top-k\text{top-}k
the sum over all documents is truncated to the kk best by retriever score (RAG: k=5k=5 or 1010)

Training maximises log⁡p(y∣x)\log p(y\mid x), and the gradient reaches the retriever through pη(z∣x)p_\eta(z\mid x): documents that make the target likely get pushed up. REALM does this during masked-LM pretraining. Since EdE_d changes, the document index goes stale, so REALM re-embeds and re-indexes the corpus asynchronously every few hundred steps. It also masks salient spans (named entities, dates) so that the prediction requires world knowledge. RAG starts from a pretrained DPR retriever and BART-large, keeps the document encoder and index fixed, and fine-tunes only the query encoder and generator.

RAG-Sequence: p(y∣x)≈∑zpη(z∣x)∏i=1Npθ(yi∣x,z,y<i)RAG-Token: p(y∣x)≈∏i=1N∑zpη(z∣x) pθ(yi∣x,z,y<i)\textbf{RAG-Sequence: } p(y\mid x)\approx\sum_{z}p_\eta(z\mid x)\prod_{i=1}^{N}p_\theta(y_i\mid x,z,y_{<i})\qquad\qquad \textbf{RAG-Token: } p(y\mid x)\approx\prod_{i=1}^{N}\sum_{z}p_\eta(z\mid x)\,p_\theta(y_i\mid x,z,y_{<i})
RAG-Sequence\text{RAG-Sequence}
one document explains the whole output; marginalise once, over complete sequences
RAG-Token\text{RAG-Token}
each output token may draw on a different document; marginalise at every step, which lets one answer combine facts from several passages

RAG-Token is an ordinary autoregressive model with a mixed next-token distribution, so standard beam search decodes it. RAG-Sequence does not factor over tokens: the paper runs beam search per document and then rescores the union of hypotheses under all documents ("thorough decoding"), or skips the extra forward passes for hypotheses a document did not produce ("fast decoding").

Fusion-in-Decoder and RETRO: retrieval through cross-attention

Fusion-in-Decoder (FiD) encodes each (question, passage) pair separately with a T5 encoder and concatenates all encoder outputs. The decoder then cross-attends to all of them at once, as in an Encoder–decoder transformer. Encoding is linear in the number of passages, since passages never attend to each other, so FiD can read 100 passages, and its accuracy keeps rising with more of them. RETRO brings retrieval into a Decoder-only transformer language model at the scale of pretraining.

C1,…,Cl  (chunks of m=64 tokens),Eu=Enc(Ret(Cu),Hu),CCA(H,E)um+i−1=CA(hum+i−1, Eu),  i=1..mC_1,\dots,C_l\ \ (\text{chunks of }m=64\text{ tokens}),\qquad E_u=\mathrm{Enc}\big(\mathrm{Ret}(C_u),H_u\big),\qquad \mathrm{CCA}(H,E)_{um+i-1}=\mathrm{CA}\big(h_{um+i-1},\,E_u\big),\ \ i=1..m
Ret(Cu)\mathrm{Ret}(C_u)
the kk nearest neighbours of chunk CuC_u in the datastore, by L2 distance between frozen BERT embeddings; each neighbour is the matching 64-token chunk plus its 64-token continuation
Enc\mathrm{Enc}
a small bidirectional Transformer over each neighbour, which cross-attends to the activations HuH_u of chunk uu
CCA\mathrm{CCA}
chunked cross-attention: the last token of chunk uu and the first m−1m-1 tokens of chunk u+1u+1 attend to the neighbours of chunk uu. No token sees neighbours retrieved with text after it, so the model stays causal

Each retrieval is done per chunk rather than per token, and the cross-attention cost grows with the number of chunks times kk neighbours of 128 tokens, so the datastore can be huge: RETRO looks up 64-token chunks in a 2-trillion-token database built from the MassiveText corpus. Because the BERT retriever is frozen, the embeddings of the whole database are computed once. The 7.5B-parameter RETRO performed comparably to GPT-3 (175B) and Jurassic-1 (178B) on the Pile, with 25 times fewer parameters. A pretrained transformer can also be RETROfitted: freeze it, add and train only the encoder and CCA layers.

Reported settings and results

kNN-LM
WikiText-103, datastore of the 103M training tokens, k=1024k=1024, λ=0.25\lambda=0.25: test perplexity 18.65 → 16.12 with no extra training
REALM
retrieval over Wikipedia split into blocks of up to 288 wordpieces; NaturalQuestions-Open exact match 40.4
RAG
DPR (BERT-base bi-encoder) + BART-large; 21M 100-word Wikipedia passages in a FAISS index; NaturalQuestions EM 44.1 (RAG-Token) and 44.5 (RAG-Sequence), vs 41.5 for the DPR extractive reader and 34.5 for closed-book T5-11B
Fusion-in-Decoder
T5-large reading 100 passages: NaturalQuestions EM 51.4, TriviaQA 67.6
RETRO
chunks of 64 tokens, neighbours of 64 + 64 tokens, frozen BERT retriever, 2T-token database; 7.5B parameters, comparable to GPT-3 and Jurassic-1 on the Pile
datastore: key = context embedding, value = next token · 68 entries● Paris  ● Rome  ● Berlin  ● Madrid  ● Vienna  ✚ test · ○ other tests (click) · ringed = new documentnext token: ▬ p_LM · ▬ p_kNN · ▬ pParis0.188Rome0.075Berlin0.338Madrid0.118Vienna ✓0.282p_LM(gold) 0.050p_kNN(gold) 0.979p(gold) 0.282 (−log p = 1.265)perplexity, all 4 test contextsLM only (λ = 0) 3.236kNN only (λ = 1) 1.914this λ = 0.25 2.075best λ 0.72 → 1.749p_kNN(y) ∝ Σ exp(−d) over the k = 8 nearest with value y · p = λ·p_kNN + (1 − λ)·p_LM
test context
datastore

The LM is frozen; adding entries to the datastore changes p_kNN without any training. d is the squared L2 distance (scaled).

fig.A toy kNN-LM. Each dot is a datastore entry: a 2-D context embedding coloured by the token that followed it. The cross is a test context; lines join it to its k nearest keys, and the bars compare p_LM, p_kNN and the mixture p. The default 'rare fact' context should be followed by Vienna. The frozen LM gives Vienna 0.050. With the 4 entries of the new document in the datastore, k = 8 and λ = 0.25, p_kNN(Vienna) = 0.979 and p = 0.282. Remove the document and the neighbours are Madrid contexts (p_kNN(Madrid) = 0.940): retrieval can only lower Vienna's probability, to 0.0375. The panel scores all 4 test contexts. With the document, the LM alone has perplexity 3.236, kNN alone 1.914, λ = 0.25 gives 2.075, and the best λ is 0.72 (1.749). The interpolation beats both because the 'unseen context' query (gold Berlin, p_LM 0.62) has mostly Madrid neighbours (p_kNN(Berlin) = 0.101). With k = 1, kNN alone gives some gold token probability 0, so its perplexity is ∞. Without the document the best λ is 0.

Worked example: RAG-Sequence vs RAG-Token

Two retrieved documents with pη(z1∣x)=0.6p_\eta(z_1\mid x)=0.6, pη(z2∣x)=0.4p_\eta(z_2\mid x)=0.4, and a two-token target y=(y1,y2)y=(y_1,y_2). Document 1 supports the first token, document 2 the second.

documentp(z | x)p(y₁ | x, z)p(y₂ | x, z, y₁)p(y | x, z)
z₁0.60.90.20.18
z₂0.40.30.90.27
RAG-Sequence: Σ p(z) p(y | z)0.216
RAG-Token: per-token mixtures0.660.480.3168
Generator probabilities of the target tokens under each document (computed in Node)

Two marginalisations, and a kNN-LM mixture

  1. 1RAG-Sequence: 0.6⋅(0.9⋅0.2)+0.4⋅(0.3⋅0.9)=0.108+0.108=0.2160.6\cdot(0.9\cdot0.2)+0.4\cdot(0.3\cdot0.9)=0.108+0.108=0.216, so −log⁡p=1.532-\log p=1.532. The posterior over documents given the whole answer is (0.5, 0.5)(0.5,\,0.5): neither explains the full answer well.
  2. 2RAG-Token, step 1: 0.6⋅0.9+0.4⋅0.3=0.660.6\cdot0.9+0.4\cdot0.3=0.66. Step 2: 0.6⋅0.2+0.4⋅0.9=0.480.6\cdot0.2+0.4\cdot0.9=0.48. Product 0.31680.3168, −log⁡p=1.149-\log p=1.149. Mixing per token lets y1y_1 come from z1z_1 and y2y_2 from z2z_2.
  3. 3RAG-Token is not always higher: if one document explained both tokens, RAG-Sequence would credit it fully, while RAG-Token would still dilute every step with the other document.
  4. 4kNN-LM with the figure's 'rare fact' context: p=0.25⋅0.979+0.75⋅0.050=0.282p=0.25\cdot0.979+0.75\cdot0.050=0.282, against 0.0500.050 for the LM alone, so −log⁡p-\log p falls from 2.996 to 1.265.
python
1import torch, torch.nn.functional as F
2
3def knn_lm(lm_logits, query, keys, values, k=1024, lam=0.25):
4 # lm_logits (V,), query (d,), keys (N, d) datastore keys, values (N,) next-token ids
5 d2 = ((keys - query) ** 2).sum(-1) # squared L2 to every key (FAISS in practice)
6 dist, idx = d2.topk(k, largest=False) # k nearest
7 w = F.softmax(-dist, dim=0) # exp(-d), normalised over the neighbours
8 p_knn = torch.zeros_like(lm_logits).scatter_add_(0, values[idx], w)
9 return lam * p_knn + (1 - lam) * F.softmax(lm_logits, dim=-1)
10
11torch.manual_seed(0)
12keys, values = torch.randn(10000, 64), torch.randint(0, 100, (10000,))
13query = keys[42] + 0.01 * torch.randn(64) # a context seen in training
14p = knn_lm(torch.randn(100), query, keys, values, k=16)
15print(p.shape, round(p.sum().item(), 4)) # torch.Size([100]) 1.0
16print((p.argmax() == values[42]).item()) # True: the stored continuation wins
17
18def rag_sequence_logp(doc_logp, tok_logp):
19 # doc_logp (K,) log p(z|x); tok_logp (K, T) log p(y_i | x, z, y_<i) of the target tokens
20 return torch.logsumexp(doc_logp + tok_logp.sum(-1), dim=0)
21
22def rag_token_logp(doc_logp, tok_logp):
23 return torch.logsumexp(doc_logp[:, None] + tok_logp, dim=0).sum()
24
25doc_logp = torch.tensor([0.6, 0.4]).log()
26tok_logp = torch.tensor([[0.9, 0.2], [0.3, 0.9]]).log()
27print(rag_sequence_logp(doc_logp, tok_logp).exp()) # tensor(0.2160)
28print(rag_token_logp(doc_logp, tok_logp).exp()) # tensor(0.3168)
kNN-LM interpolation and the two RAG marginalisations

Use when

  • +Knowledge can be updated, extended or removed by editing the index, without retraining
  • +Small models with retrieval match much larger closed-book models (RETRO 7.5B vs GPT-3 175B on the Pile)
  • +Retrieved passages give provenance: you can show which documents an answer used
  • +Large gains on rare entities and long-tail facts

Avoid when

  • −An index of billions of embeddings costs memory and latency; approximate search can miss neighbours
  • −Joint training is hard: document embeddings go stale (REALM re-indexes) or are frozen (RAG, RETRO)
  • −Bad or adversarial retrieved text is copied confidently
  • −Benchmark gains can come from leakage between datastore and test set

Check yourself

A kNN-LM's datastore has no entry whose value is the correct next token, and λ = 1. What is the model's perplexity on that token?

Check yourself

When does RAG-Token assign a target answer a higher probability than RAG-Sequence?

Remember

  • ✓kNN-LM: p=λ pkNN+(1−λ) pLMp=\lambda\,p_{\text{kNN}}+(1-\lambda)\,p_{\text{LM}}, with pkNN(y)∝∑neighbours with value yexp⁡(−d)p_{\text{kNN}}(y)\propto\sum_{\text{neighbours with value }y}\exp(-d) over stored (hidden state, next token) pairs; WikiText-103 perplexity 18.65 → 16.12 with no training.
  • ✓RAG: p(y∣x)≈∑z∈top-kpη(z∣x) pθ(y∣x,z)p(y\mid x)\approx\sum_{z\in\text{top-}k}p_\eta(z\mid x)\,p_\theta(y\mid x,z); RAG-Sequence marginalises over whole outputs, RAG-Token at every token.
  • ✓REALM trains the dense retriever during masked-LM pretraining, re-indexing the corpus asynchronously because document embeddings change.
  • ✓Fusion-in-Decoder encodes each passage separately and lets the decoder cross-attend to all of them (100 passages, NQ EM 51.4).
  • ✓RETRO retrieves neighbours per 64-token chunk from a 2T-token database with a frozen BERT retriever and reads them by chunked cross-attention; 7.5B RETRO is comparable to GPT-3 on the Pile.
← back to Deep Learning roadmap