# Retrieval-augmented transformers

> A retrieval-augmented transformer looks up text or stored hidden states in an external datastore and conditions its prediction on them: kNN-LM mixes a neighbour distribution into the output, REALM and RAG marginalise over retrieved passages, RETRO cross-attends to retrieved chunks.

Source: https://www.mlfoundry.in/learn/retrieval-augmented-transformers  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 16 min read · updated 2026-10-12

**Retrieval-augmented transformer:** is a Transformer language model that, for each input, **retrieves** items from a large external datastore (passages, chunks, or stored (context, next-token) pairs) by nearest-neighbour search over learned embeddings, and **conditions** its prediction on them. Knowledge then lives partly in an index that can be inspected, replaced or extended without retraining, rather than only in the weights. The main designs differ in *where* retrieval enters: at the output distribution (kNN-LM), as a latent variable that is marginalised (REALM, RAG), or through **Cross-attention** inside the network (Fusion-in-Decoder, RETRO).

Closed-book language models store facts in their parameters. Updating a fact means retraining, the model cannot point to a source, and rare facts need very large models. Open-domain QA systems had long used a retrieve-then-read pipeline (Chen et al.'s DrQA, 2017) with a fixed retriever such as TF-IDF. In 2020 several papers made retrieval part of the neural model. Khandelwal, Levy, Jurafsky, Zettlemoyer and Lewis (**kNN-LM**, ICLR 2020) interpolated a pretrained LM with nearest neighbours in its own hidden-state space. Guu, Lee, Tung, Pasupat and Chang (**REALM**, ICML 2020) trained a dense retriever jointly with a masked language model. Lewis et al. (**RAG**, NeurIPS 2020) combined a dense retriever with a seq2seq generator. Izacard and Grave's **Fusion-in-Decoder** (EACL 2021) and Borgeaud et al.'s **RETRO** (ICML 2022) moved retrieved text into cross-attention, RETRO with a datastore of 2 trillion tokens.

## kNN-LM: retrieval at the output

$$
p_{\text{kNN}}(y\mid x)\ \propto \sum_{(k_i,v_i)\in\mathcal N_k(x)} \mathbb 1[y=v_i]\,\exp\!\big(-d(k_i,f(x))\big),\qquad p(y\mid x)=\lambda\,p_{\text{kNN}}(y\mid x)+(1-\lambda)\,p_{\text{LM}}(y\mid x)
$$

- $f(x)$: the LM's hidden representation of the context $x$ (in the paper the 1024-d input to the final feed-forward layer)
- $(k_i,v_i)$: datastore entry: key $f(c_i)$ for a training context $c_i$, value the token $w_i$ that followed it; one entry per training token
- $\mathcal N_k(x)$: the $k$ keys nearest to $f(x)$, found with an approximate index (FAISS)
- $d$: squared Euclidean distance (it worked better than the inner product)
- $\lambda$: interpolation weight tuned on validation data (0.25 for WikiText-103)

No parameters are trained: the datastore is one forward pass over the training data. The neighbours help most on rare patterns such as names and factual phrases, which the softmax layer generalises poorly but which appear verbatim in similar contexts. Because the datastore is separate from the model, a model trained on one corpus can be given the datastore of another, in-domain one, and its perplexity there drops without fine-tuning.

## REALM and RAG: retrieved passages as a latent variable

$$
p_\eta(z\mid x)=\frac{\exp\big(E_d(z)^\top E_q(x)\big)}{\sum_{z'}\exp\big(E_d(z')^\top E_q(x)\big)},\qquad p(y\mid x)\approx\sum_{z\in\text{top-}k}p_\eta(z\mid x)\,p_\theta(y\mid x,z)
$$

- $E_q,E_d$: query and document encoders (BERT-based bi-encoders); documents are embedded once and indexed for maximum inner product search (MIPS)
- $z$: a retrieved passage, treated as a latent variable: there is no label saying which passage is right
- $p_\theta$: the reader: a masked LM in REALM, a BART seq2seq generator in RAG
- $\text{top-}k$: the sum over all documents is truncated to the $k$ best by retriever score (RAG: $k=5$ or $10$)

Training maximises $\log p(y\mid x)$, and the gradient reaches the retriever through $p_\eta(z\mid x)$: documents that make the target likely get pushed up. **REALM** does this during masked-LM pretraining. Since $E_d$ changes, the document index goes stale, so REALM re-embeds and re-indexes the corpus asynchronously every few hundred steps. It also masks *salient spans* (named entities, dates) so that the prediction requires world knowledge. **RAG** starts from a pretrained DPR retriever and BART-large, keeps the document encoder and index fixed, and fine-tunes only the query encoder and generator.

$$
\textbf{RAG-Sequence: } p(y\mid x)\approx\sum_{z}p_\eta(z\mid x)\prod_{i=1}^{N}p_\theta(y_i\mid x,z,y_{<i})\qquad\qquad \textbf{RAG-Token: } p(y\mid x)\approx\prod_{i=1}^{N}\sum_{z}p_\eta(z\mid x)\,p_\theta(y_i\mid x,z,y_{<i})
$$

- $\text{RAG-Sequence}$: one document explains the whole output; marginalise once, over complete sequences
- $\text{RAG-Token}$: each output token may draw on a different document; marginalise at every step, which lets one answer combine facts from several passages

RAG-Token is an ordinary autoregressive model with a mixed next-token distribution, so standard beam search decodes it. RAG-Sequence does not factor over tokens: the paper runs beam search per document and then rescores the union of hypotheses under all documents ("thorough decoding"), or skips the extra forward passes for hypotheses a document did not produce ("fast decoding").

## Fusion-in-Decoder and RETRO: retrieval through cross-attention

**Fusion-in-Decoder (FiD)** encodes each (question, passage) pair separately with a T5 encoder and concatenates all encoder outputs. The decoder then cross-attends to all of them at once, as in an **Encoder–decoder transformer**. Encoding is linear in the number of passages, since passages never attend to each other, so FiD can read 100 passages, and its accuracy keeps rising with more of them. **RETRO** brings retrieval into a **Decoder-only transformer** language model at the scale of pretraining.

$$
C_1,\dots,C_l\ \ (\text{chunks of }m=64\text{ tokens}),\qquad E_u=\mathrm{Enc}\big(\mathrm{Ret}(C_u),H_u\big),\qquad \mathrm{CCA}(H,E)_{um+i-1}=\mathrm{CA}\big(h_{um+i-1},\,E_u\big),\ \ i=1..m
$$

- $\mathrm{Ret}(C_u)$: the $k$ nearest neighbours of chunk $C_u$ in the datastore, by L2 distance between frozen BERT embeddings; each neighbour is the matching 64-token chunk plus its 64-token continuation
- $\mathrm{Enc}$: a small bidirectional Transformer over each neighbour, which cross-attends to the activations $H_u$ of chunk $u$
- $\mathrm{CCA}$: chunked cross-attention: the last token of chunk $u$ and the first $m-1$ tokens of chunk $u+1$ attend to the neighbours of chunk $u$. No token sees neighbours retrieved with text after it, so the model stays causal

Each retrieval is done per chunk rather than per token, and the cross-attention cost grows with the number of chunks times $k$ neighbours of 128 tokens, so the datastore can be huge: RETRO looks up 64-token chunks in a 2-trillion-token database built from the MassiveText corpus. Because the BERT retriever is frozen, the embeddings of the whole database are computed once. The 7.5B-parameter RETRO performed comparably to GPT-3 (175B) and Jurassic-1 (178B) on the Pile, with 25 times fewer parameters. A pretrained transformer can also be **RETROfitted**: freeze it, add and train only the encoder and CCA layers.

**Reported settings and results**

- **kNN-LM:** WikiText-103, datastore of the 103M training tokens, $k=1024$, $\lambda=0.25$: test perplexity 18.65 → 16.12 with no extra training
- **REALM:** retrieval over Wikipedia split into blocks of up to 288 wordpieces; NaturalQuestions-Open exact match 40.4
- **RAG:** DPR (BERT-base bi-encoder) + BART-large; 21M 100-word Wikipedia passages in a FAISS index; NaturalQuestions EM 44.1 (RAG-Token) and 44.5 (RAG-Sequence), vs 41.5 for the DPR extractive reader and 34.5 for closed-book T5-11B
- **Fusion-in-Decoder:** T5-large reading 100 passages: NaturalQuestions EM 51.4, TriviaQA 67.6
- **RETRO:** chunks of 64 tokens, neighbours of 64 + 64 tokens, frozen BERT retriever, 2T-token database; 7.5B parameters, comparable to GPT-3 and Jurassic-1 on the Pile

> Figure: A toy kNN-LM. Each dot is a datastore entry: a 2-D context embedding coloured by the token that followed it. The cross is a test context; lines join it to its k nearest keys, and the bars compare p_LM, p_kNN and the mixture p. The default 'rare fact' context should be followed by Vienna. The frozen LM gives Vienna 0.050. With the 4 entries of the new document in the datastore, k = 8 and λ = 0.25, p_kNN(Vienna) = 0.979 and p = 0.282. Remove the document and the neighbours are Madrid contexts (p_kNN(Madrid) = 0.940): retrieval can only lower Vienna's probability, to 0.0375. The panel scores all 4 test contexts. With the document, the LM alone has perplexity 3.236, kNN alone 1.914, λ = 0.25 gives 2.075, and the best λ is 0.72 (1.749). The interpolation beats both because the 'unseen context' query (gold Berlin, p_LM 0.62) has mostly Madrid neighbours (p_kNN(Berlin) = 0.101). With k = 1, kNN alone gives some gold token probability 0, so its perplexity is ∞. Without the document the best λ is 0.

## Worked example: RAG-Sequence vs RAG-Token

Two retrieved documents with $p_\eta(z_1\mid x)=0.6$, $p_\eta(z_2\mid x)=0.4$, and a two-token target $y=(y_1,y_2)$. Document 1 supports the first token, document 2 the second.

Generator probabilities of the target tokens under each document (computed in Node)

| document | p(z \| x) | p(y₁ \| x, z) | p(y₂ \| x, z, y₁) | p(y \| x, z) |
| --- | --- | --- | --- | --- |
| z₁ | 0.6 | 0.9 | 0.2 | 0.18 |
| z₂ | 0.4 | 0.3 | 0.9 | 0.27 |
| RAG-Sequence: Σ p(z) p(y \| z) |  |  |  | 0.216 |
| RAG-Token: per-token mixtures |  | 0.66 | 0.48 | 0.3168 |

**Two marginalisations, and a kNN-LM mixture**

1. RAG-Sequence: $0.6\cdot(0.9\cdot0.2)+0.4\cdot(0.3\cdot0.9)=0.108+0.108=0.216$, so $-\log p=1.532$. The posterior over documents given the whole answer is $(0.5,\,0.5)$: neither explains the full answer well.
2. RAG-Token, step 1: $0.6\cdot0.9+0.4\cdot0.3=0.66$. Step 2: $0.6\cdot0.2+0.4\cdot0.9=0.48$. Product $0.3168$, $-\log p=1.149$. Mixing per token lets $y_1$ come from $z_1$ and $y_2$ from $z_2$.
3. RAG-Token is not always higher: if one document explained both tokens, RAG-Sequence would credit it fully, while RAG-Token would still dilute every step with the other document.
4. kNN-LM with the figure's 'rare fact' context: $p=0.25\cdot0.979+0.75\cdot0.050=0.282$, against $0.050$ for the LM alone, so $-\log p$ falls from 2.996 to 1.265.

```python
import torch, torch.nn.functional as F

def knn_lm(lm_logits, query, keys, values, k=1024, lam=0.25):
    # lm_logits (V,), query (d,), keys (N, d) datastore keys, values (N,) next-token ids
    d2 = ((keys - query) ** 2).sum(-1)                  # squared L2 to every key (FAISS in practice)
    dist, idx = d2.topk(k, largest=False)              # k nearest
    w = F.softmax(-dist, dim=0)                        # exp(-d), normalised over the neighbours
    p_knn = torch.zeros_like(lm_logits).scatter_add_(0, values[idx], w)
    return lam * p_knn + (1 - lam) * F.softmax(lm_logits, dim=-1)

torch.manual_seed(0)
keys, values = torch.randn(10000, 64), torch.randint(0, 100, (10000,))
query = keys[42] + 0.01 * torch.randn(64)              # a context seen in training
p = knn_lm(torch.randn(100), query, keys, values, k=16)
print(p.shape, round(p.sum().item(), 4))               # torch.Size([100]) 1.0
print((p.argmax() == values[42]).item())               # True: the stored continuation wins

def rag_sequence_logp(doc_logp, tok_logp):
    # doc_logp (K,) log p(z|x); tok_logp (K, T) log p(y_i | x, z, y_<i) of the target tokens
    return torch.logsumexp(doc_logp + tok_logp.sum(-1), dim=0)

def rag_token_logp(doc_logp, tok_logp):
    return torch.logsumexp(doc_logp[:, None] + tok_logp, dim=0).sum()

doc_logp = torch.tensor([0.6, 0.4]).log()
tok_logp = torch.tensor([[0.9, 0.2], [0.3, 0.9]]).log()
print(rag_sequence_logp(doc_logp, tok_logp).exp())    # tensor(0.2160)
print(rag_token_logp(doc_logp, tok_logp).exp())       # tensor(0.3168)
```

kNN-LM interpolation and the two RAG marginalisations

> **Test-set leakage inflates retrieval gains:** If the datastore contains the evaluation text, or near-duplicates of it, retrieval copies the answer, and perplexity or QA accuracy looks much better than it should. With trillions of web tokens this is likely, not hypothetical. The RETRO paper measures how much of each evaluation chunk overlaps with retrieved neighbours and reports results as a function of that overlap. Deduplicate the datastore against the test set, or report the overlap.

> **Where retrieval enters decides what it can do:** kNN-LM changes only the output distribution. It can copy a token seen after a similar context, but cannot reason over a retrieved passage. REALM and RAG let the reader process whole passages, but each passage is read in isolation and combined only through the mixture weights. FiD and RETRO cross-attend to all retrieved text at once, so evidence from several passages can be combined inside the network. The price is that the model has to be trained, or RETROfitted, to use it.

> **Retrieval is not the same as a long context:** A long-context model such as **Longformer** or **Transformer-XL** attends to text it was given. A retrieval model *chooses* what to read from a corpus far larger than any context window, with a separate similarity search. The two combine: FiD-style readers often use efficient attention to read more passages, and modern retrieval-augmented generation pipelines put retrieved passages into the prompt of a **Decoder-only transformer**.

**Strengths**

- Knowledge can be updated, extended or removed by editing the index, without retraining
- Small models with retrieval match much larger closed-book models (RETRO 7.5B vs GPT-3 175B on the Pile)
- Retrieved passages give provenance: you can show which documents an answer used
- Large gains on rare entities and long-tail facts

**Limitations**

- An index of billions of embeddings costs memory and latency; approximate search can miss neighbours
- Joint training is hard: document embeddings go stale (REALM re-indexes) or are frozen (RAG, RETRO)
- Bad or adversarial retrieved text is copied confidently
- Benchmark gains can come from leakage between datastore and test set

**Question:** A kNN-LM's datastore has no entry whose value is the correct next token, and λ = 1. What is the model's perplexity on that token?

- A. Infinite: p_kNN gives the token probability 0
- B. The same as the base LM
- C. Lower than the base LM
- D. It depends only on k

**Answer:** A. Infinite: p_kNN gives the token probability 0 — $p_{\text{kNN}}$ only puts mass on values of retrieved neighbours, so the token gets probability exactly 0 and $-\log p=\infty$. Interpolating with $\lambda<1$ keeps $(1-\lambda)\,p_{\text{LM}}>0$, which is one reason the mixture is used rather than the neighbours alone.

**Question:** When does RAG-Token assign a target answer a higher probability than RAG-Sequence?

- A. Always, because it marginalises more often
- B. When a single document supports the whole answer
- C. When different parts of the answer are supported by different retrieved documents
- D. Never; they are equal by definition

**Answer:** C. When different parts of the answer are supported by different retrieved documents — RAG-Token mixes documents at each step, so token 1 can come from $z_1$ and token 2 from $z_2$ (0.3168 vs 0.216 in the worked example). If one document explains everything, RAG-Sequence credits that document's full product, while RAG-Token keeps mixing in the others at every step.

**Key points**

- kNN-LM: $p=\lambda\,p_{\text{kNN}}+(1-\lambda)\,p_{\text{LM}}$, with $p_{\text{kNN}}(y)\propto\sum_{\text{neighbours with value }y}\exp(-d)$ over stored (hidden state, next token) pairs; WikiText-103 perplexity 18.65 → 16.12 with no training.
- RAG: $p(y\mid x)\approx\sum_{z\in\text{top-}k}p_\eta(z\mid x)\,p_\theta(y\mid x,z)$; RAG-Sequence marginalises over whole outputs, RAG-Token at every token.
- REALM trains the dense retriever during masked-LM pretraining, re-indexing the corpus asynchronously because document embeddings change.
- Fusion-in-Decoder encodes each passage separately and lets the decoder cross-attend to all of them (100 passages, NQ EM 51.4).
- RETRO retrieves neighbours per 64-token chunk from a 2T-token database with a frozen BERT retriever and reads them by chunked cross-attention; 7.5B RETRO is comparable to GPT-3 on the Pile.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
