Definition
Retrieval-augmented transformer is a Transformer language model that, for each input, retrieves items from a large external datastore (passages, chunks, or stored (context, next-token) pairs) by nearest-neighbour search over learned embeddings, and conditions its prediction on them. Knowledge then lives partly in an index that can be inspected, replaced or extended without retraining, rather than only in the weights. The main designs differ in where retrieval enters: at the output distribution (kNN-LM), as a latent variable that is marginalised (REALM, RAG), or through Cross-attention inside the network (Fusion-in-Decoder, RETRO).
Closed-book language models store facts in their parameters. Updating a fact means retraining, the model cannot point to a source, and rare facts need very large models. Open-domain QA systems had long used a retrieve-then-read pipeline (Chen et al.'s DrQA, 2017) with a fixed retriever such as TF-IDF. In 2020 several papers made retrieval part of the neural model. Khandelwal, Levy, Jurafsky, Zettlemoyer and Lewis (kNN-LM, ICLR 2020) interpolated a pretrained LM with nearest neighbours in its own hidden-state space. Guu, Lee, Tung, Pasupat and Chang (REALM, ICML 2020) trained a dense retriever jointly with a masked language model. Lewis et al. (RAG, NeurIPS 2020) combined a dense retriever with a seq2seq generator. Izacard and Grave's Fusion-in-Decoder (EACL 2021) and Borgeaud et al.'s RETRO (ICML 2022) moved retrieved text into cross-attention, RETRO with a datastore of 2 trillion tokens.
kNN-LM: retrieval at the output
- the LM's hidden representation of the context (in the paper the 1024-d input to the final feed-forward layer)
- datastore entry: key for a training context , value the token that followed it; one entry per training token
- the keys nearest to , found with an approximate index (FAISS)
- squared Euclidean distance (it worked better than the inner product)
- interpolation weight tuned on validation data (0.25 for WikiText-103)
No parameters are trained: the datastore is one forward pass over the training data. The neighbours help most on rare patterns such as names and factual phrases, which the softmax layer generalises poorly but which appear verbatim in similar contexts. Because the datastore is separate from the model, a model trained on one corpus can be given the datastore of another, in-domain one, and its perplexity there drops without fine-tuning.
REALM and RAG: retrieved passages as a latent variable
- query and document encoders (BERT-based bi-encoders); documents are embedded once and indexed for maximum inner product search (MIPS)
- a retrieved passage, treated as a latent variable: there is no label saying which passage is right
- the reader: a masked LM in REALM, a BART seq2seq generator in RAG
- the sum over all documents is truncated to the best by retriever score (RAG: or )
Training maximises , and the gradient reaches the retriever through : documents that make the target likely get pushed up. REALM does this during masked-LM pretraining. Since changes, the document index goes stale, so REALM re-embeds and re-indexes the corpus asynchronously every few hundred steps. It also masks salient spans (named entities, dates) so that the prediction requires world knowledge. RAG starts from a pretrained DPR retriever and BART-large, keeps the document encoder and index fixed, and fine-tunes only the query encoder and generator.
- one document explains the whole output; marginalise once, over complete sequences
- each output token may draw on a different document; marginalise at every step, which lets one answer combine facts from several passages
RAG-Token is an ordinary autoregressive model with a mixed next-token distribution, so standard beam search decodes it. RAG-Sequence does not factor over tokens: the paper runs beam search per document and then rescores the union of hypotheses under all documents ("thorough decoding"), or skips the extra forward passes for hypotheses a document did not produce ("fast decoding").
Fusion-in-Decoder and RETRO: retrieval through cross-attention
Fusion-in-Decoder (FiD) encodes each (question, passage) pair separately with a T5 encoder and concatenates all encoder outputs. The decoder then cross-attends to all of them at once, as in an Encoder–decoder transformer. Encoding is linear in the number of passages, since passages never attend to each other, so FiD can read 100 passages, and its accuracy keeps rising with more of them. RETRO brings retrieval into a Decoder-only transformer language model at the scale of pretraining.
- the nearest neighbours of chunk in the datastore, by L2 distance between frozen BERT embeddings; each neighbour is the matching 64-token chunk plus its 64-token continuation
- a small bidirectional Transformer over each neighbour, which cross-attends to the activations of chunk
- chunked cross-attention: the last token of chunk and the first tokens of chunk attend to the neighbours of chunk . No token sees neighbours retrieved with text after it, so the model stays causal
Each retrieval is done per chunk rather than per token, and the cross-attention cost grows with the number of chunks times neighbours of 128 tokens, so the datastore can be huge: RETRO looks up 64-token chunks in a 2-trillion-token database built from the MassiveText corpus. Because the BERT retriever is frozen, the embeddings of the whole database are computed once. The 7.5B-parameter RETRO performed comparably to GPT-3 (175B) and Jurassic-1 (178B) on the Pile, with 25 times fewer parameters. A pretrained transformer can also be RETROfitted: freeze it, add and train only the encoder and CCA layers.
Reported settings and results
- kNN-LM
- WikiText-103, datastore of the 103M training tokens, , : test perplexity 18.65 → 16.12 with no extra training
- REALM
- retrieval over Wikipedia split into blocks of up to 288 wordpieces; NaturalQuestions-Open exact match 40.4
- RAG
- DPR (BERT-base bi-encoder) + BART-large; 21M 100-word Wikipedia passages in a FAISS index; NaturalQuestions EM 44.1 (RAG-Token) and 44.5 (RAG-Sequence), vs 41.5 for the DPR extractive reader and 34.5 for closed-book T5-11B
- Fusion-in-Decoder
- T5-large reading 100 passages: NaturalQuestions EM 51.4, TriviaQA 67.6
- RETRO
- chunks of 64 tokens, neighbours of 64 + 64 tokens, frozen BERT retriever, 2T-token database; 7.5B parameters, comparable to GPT-3 and Jurassic-1 on the Pile
The LM is frozen; adding entries to the datastore changes p_kNN without any training. d is the squared L2 distance (scaled).
Worked example: RAG-Sequence vs RAG-Token
Two retrieved documents with , , and a two-token target . Document 1 supports the first token, document 2 the second.
| document | p(z | x) | p(y₁ | x, z) | p(y₂ | x, z, y₁) | p(y | x, z) |
|---|---|---|---|---|
| z₁ | 0.6 | 0.9 | 0.2 | 0.18 |
| z₂ | 0.4 | 0.3 | 0.9 | 0.27 |
| RAG-Sequence: Σ p(z) p(y | z) | 0.216 | |||
| RAG-Token: per-token mixtures | 0.66 | 0.48 | 0.3168 |
Two marginalisations, and a kNN-LM mixture
- 1RAG-Sequence: , so . The posterior over documents given the whole answer is : neither explains the full answer well.
- 2RAG-Token, step 1: . Step 2: . Product , . Mixing per token lets come from and from .
- 3RAG-Token is not always higher: if one document explained both tokens, RAG-Sequence would credit it fully, while RAG-Token would still dilute every step with the other document.
- 4kNN-LM with the figure's 'rare fact' context: , against for the LM alone, so falls from 2.996 to 1.265.
1import torch, torch.nn.functional as F23def knn_lm(lm_logits, query, keys, values, k=1024, lam=0.25):4 # lm_logits (V,), query (d,), keys (N, d) datastore keys, values (N,) next-token ids5 d2 = ((keys - query) ** 2).sum(-1) # squared L2 to every key (FAISS in practice)6 dist, idx = d2.topk(k, largest=False) # k nearest7 w = F.softmax(-dist, dim=0) # exp(-d), normalised over the neighbours8 p_knn = torch.zeros_like(lm_logits).scatter_add_(0, values[idx], w)9 return lam * p_knn + (1 - lam) * F.softmax(lm_logits, dim=-1)1011torch.manual_seed(0)12keys, values = torch.randn(10000, 64), torch.randint(0, 100, (10000,))13query = keys[42] + 0.01 * torch.randn(64) # a context seen in training14p = knn_lm(torch.randn(100), query, keys, values, k=16)15print(p.shape, round(p.sum().item(), 4)) # torch.Size([100]) 1.016print((p.argmax() == values[42]).item()) # True: the stored continuation wins1718def rag_sequence_logp(doc_logp, tok_logp):19 # doc_logp (K,) log p(z|x); tok_logp (K, T) log p(y_i | x, z, y_<i) of the target tokens20 return torch.logsumexp(doc_logp + tok_logp.sum(-1), dim=0)2122def rag_token_logp(doc_logp, tok_logp):23 return torch.logsumexp(doc_logp[:, None] + tok_logp, dim=0).sum()2425doc_logp = torch.tensor([0.6, 0.4]).log()26tok_logp = torch.tensor([[0.9, 0.2], [0.3, 0.9]]).log()27print(rag_sequence_logp(doc_logp, tok_logp).exp()) # tensor(0.2160)28print(rag_token_logp(doc_logp, tok_logp).exp()) # tensor(0.3168)Use when
- +Knowledge can be updated, extended or removed by editing the index, without retraining
- +Small models with retrieval match much larger closed-book models (RETRO 7.5B vs GPT-3 175B on the Pile)
- +Retrieved passages give provenance: you can show which documents an answer used
- +Large gains on rare entities and long-tail facts
Avoid when
- −An index of billions of embeddings costs memory and latency; approximate search can miss neighbours
- −Joint training is hard: document embeddings go stale (REALM re-indexes) or are frozen (RAG, RETRO)
- −Bad or adversarial retrieved text is copied confidently
- −Benchmark gains can come from leakage between datastore and test set
Check yourself
A kNN-LM's datastore has no entry whose value is the correct next token, and λ = 1. What is the model's perplexity on that token?
Check yourself
When does RAG-Token assign a target answer a higher probability than RAG-Sequence?
Remember
- ✓kNN-LM: , with over stored (hidden state, next token) pairs; WikiText-103 perplexity 18.65 → 16.12 with no training.
- ✓RAG: ; RAG-Sequence marginalises over whole outputs, RAG-Token at every token.
- ✓REALM trains the dense retriever during masked-LM pretraining, re-indexing the corpus asynchronously because document embeddings change.
- ✓Fusion-in-Decoder encodes each passage separately and lets the decoder cross-attend to all of them (100 passages, NQ EM 51.4).
- ✓RETRO retrieves neighbours per 64-token chunk from a 2T-token database with a frozen BERT retriever and reads them by chunked cross-attention; 7.5B RETRO is comparable to GPT-3 on the Pile.