Definition
Transformer decoder is the half of the Transformer that generates the output sequence. It models . In the original encoder–decoder model each of its identical layers has three sublayers: masked multi-head self-attention over the target positions, in which position may only look at positions ; cross-attention whose queries come from the decoder and whose keys and values come from the encoder output; and a position-wise feed-forward network. Each sublayer is wrapped in a residual connection and LayerNorm. A final linear layer and softmax give the distribution of the next token.
The decoder was introduced with the encoder in Vaswani et al., "Attention Is All You Need" (NeurIPS 2017; see Encoder–decoder transformer). It replaced the recurrent decoders of Bahdanau attention and Luong attention, where attention was an add-on to an LSTM. Here the decoder itself is built from attention. Two properties had to be kept from the RNN decoders. The model must be autoregressive: the prediction for may depend only on and . It must also stay trainable in parallel. The causal mask does both: all target positions are computed in one pass, and none of them can see the future.
Remove the cross-attention sublayer and the decoder becomes a language model of alone. That is the Decoder-only transformer of GPT and its successors. This article covers the decoder as part of the encoder–decoder model, where it is conditioned on a source sequence.
One layer
- layer input; for layer 1 the embeddings of the shifted-right targets , times , plus positional encodings
- score set to for , so row is a softmax over positions only (see Causal attention)
- the encoder's final output (the "memory"); the same is read by every decoder layer
- Cross-attention: queries , keys , values ; a weight matrix with no causal mask
- the token embedding matrix; the 2017 model shares it between both embedding layers and the output projection
- post-LN as in the original; pre-LN decoders apply LN to each sublayer input and once after the stack (see Transformer encoder)
The order matters. Masked self-attention first collects what has been generated so far into each position. Cross-attention then uses that summary as a query, asking which source tokens are relevant for the next output. The FFN processes the result per position. Compared with an encoder layer, a decoder layer has one more attention block: parameters instead of . For that is 4,204,032 against 3,152,384.
Training: all positions at once
With teacher forcing (see Teacher forcing) the decoder input is the gold target shifted right by one, and the training target at position is . All positions are computed in one forward pass, with one masked self-attention per head and one cross-attention. The loss is the sum of cross-entropies. An RNN decoder needs sequential steps for the same work.
- start token; the last target is the end token ⟨/s⟩
- row of the output softmax; because of the mask it cannot depend on
The mask is what makes the shift honest. Decoder input position is the target of position . Without the mask, row can attend to it and copy the answer. The training loss then drops towards zero, and the model is useless at inference, where that token does not exist yet. Vaswani et al. also used label smoothing with , which in their words "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score".
Inference: one token per step
At test time there is no gold target. The encoder runs once. The decoder starts from , computes , picks a token (greedy, sampling or Beam search), appends it and repeats until ⟨/s⟩. Done naively, step re-runs the decoder on the whole prefix of length . Because of the causal mask, the states of positions are the same as at the previous step, so they can be stored instead of recomputed:
- the KV cache of layer : keys and values of all earlier positions, numbers per layer
- only the newest position needs a query; every step processes a single token per layer
- the cross-attention keys and values depend only on the encoder output; they are computed once per layer and reused at every step
With the cache, generating tokens costs new key/value projections and query–key products per layer and head. Re-running the prefix costs projections and products. For that is 5,050 against 338,350 products. The price is memory. The cache holds numbers per generated token, plus the fixed of the cross-attention memory, and it is multiplied by the batch size and the beam width.
The decoder in Vaswani et al. (2017)
- Stack
- layers; base , , 8 heads; big , , 16 heads
- Sublayers
- masked self-attention, encoder–decoder attention, FFN; each LN(x + Dropout(Sublayer(x)))
- Embeddings
- one matrix shared by the two embedding layers and the pre-softmax projection; embeddings multiplied by
- Regularisation
- residual dropout (base), label smoothing
- Decoding
- beam search with beam size 4 and length penalty ; maximum output length = input length + 50, stopping early when possible
- Parameters per layer
- : 4,204,032 (base), 16,796,672 (big)
Mask-off is a training-only bug: at inference the future does not exist yet.
Worked example: a masked row and the cost of decoding
Decoder position 2 (query) has scaled scores against keys 0, 1, 2 and against key 3, a future position. Values , , .
| key j | score | exp | weight, causal | weight, no mask |
|---|---|---|---|---|
| 0 | 1.0 | 2.7183 | 0.2399 | 0.0865 |
| 1 | 0.2 | 1.2214 | 0.1078 | 0.0389 |
| 2 | 2.0 | 7.3891 | 0.6522 | 0.2352 |
| 3 (future) | 3.0 | 20.0855 | 0 (masked) | 0.6394 |
From weights to output, then to cost
- 1Causal: the normaliser is , and the output is .
- 2Without the mask the normaliser is and 64% of the weight goes to position 3. In teacher forcing that is the input token , the very token position 2 is trained to predict.
- 3Cost of generating tokens, per layer and head: with a KV cache, 100 key/value projections and products; re-running the prefix, projections and products, 67 times as many.
- 4Memory for Transformer big (, , fp16): cached values, 24 KiB, per token. 100 tokens take 2.34 MiB per sequence, and 9.38 MiB for a beam of 4, plus the cross-attention keys and values of the source.
1import torch, torch.nn as nn, torch.nn.functional as F23class DecoderLayer(nn.Module):4 def __init__(self, d=512, h=8, d_ff=2048, p=0.1):5 super().__init__()6 self.h = h7 self.qkv, self.o1 = nn.Linear(d, 3 * d), nn.Linear(d, d) # masked self-attention8 self.q2, self.kv2, self.o2 = nn.Linear(d, d), nn.Linear(d, 2 * d), nn.Linear(d, d) # cross9 self.ff = nn.Sequential(nn.Linear(d, d_ff), nn.ReLU(), nn.Dropout(p), nn.Linear(d_ff, d))10 self.n1, self.n2, self.n3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d)11 self.drop = nn.Dropout(p)1213 def split(self, x): # (N, L, d) -> (N, h, L, d/h)14 return x.unflatten(-1, (self.h, -1)).transpose(1, 2)1516 def merge(self, x): # (N, h, L, d/h) -> (N, L, d)17 return x.transpose(1, 2).flatten(-2)1819 def forward(self, y, mem, mem_pad=None, cache=None):20 # y (N, T_new, d): new decoder positions; mem (N, S, d): encoder output21 q, k, v = map(self.split, self.qkv(y).chunk(3, -1))22 if cache is not None:23 if "k" in cache: # append to the cached keys/values24 k, v = torch.cat([cache["k"], k], 2), torch.cat([cache["v"], v], 2)25 cache["k"], cache["v"] = k, v26 t_new, t_all = q.size(2), k.size(2)27 keep = torch.ones(t_new, t_all, dtype=torch.bool, device=y.device).tril(t_all - t_new)28 a = F.scaled_dot_product_attention(q, k, v, attn_mask=keep) # True = may attend29 y = self.n1(y + self.drop(self.o1(self.merge(a))))3031 if cache is not None and "mk" in cache: # cross K, V: computed once per sequence32 mk, mv = cache["mk"], cache["mv"]33 else:34 mk, mv = map(self.split, self.kv2(mem).chunk(2, -1))35 if cache is not None:36 cache["mk"], cache["mv"] = mk, mv37 keep_src = None if mem_pad is None else ~mem_pad[:, None, None, :]38 a = F.scaled_dot_product_attention(self.split(self.q2(y)), mk, mv, attn_mask=keep_src)39 y = self.n2(y + self.drop(self.o2(self.merge(a))))40 return self.n3(y + self.drop(self.ff(y)))4142torch.manual_seed(0)43layer = DecoderLayer(d=64, h=4, d_ff=256).eval()44mem, y = torch.randn(2, 7, 64), torch.randn(2, 5, 64)4546full = layer(y, mem) # training-style: all 5 positions at once47cache = {}48steps = [layer(y[:, t:t + 1], mem, cache=cache) for t in range(5)] # inference-style49print(full.shape, torch.allclose(full, torch.cat(steps, 1), atol=1e-5)) # torch.Size([2, 5, 64]) True50print(cache["k"].shape, cache["mk"].shape) # torch.Size([2, 4, 5, 16]) torch.Size([2, 4, 7, 16])51print(sum(p.numel() for p in DecoderLayer().parameters())) # 4204032 = 16*512**2 + 19*51252print(sum(p.numel() for p in nn.TransformerDecoderLayer(512, 8, 2048).parameters())) # 4204032Use when
- +Training is parallel over all target positions: one masked pass instead of T recurrent steps
- +Every layer cross-attends to the full encoder memory, so no fixed-size bottleneck vector
- +Exact autoregressive factorisation: likelihoods, sampling and beam search all work directly
- +A KV cache makes each generation step cost O(t·d) in attention instead of re-running the prefix
Avoid when
- −Generation is still sequential: T tokens need T decoder passes
- −The KV cache grows as values per token per sequence, times batch and beam
- −Teacher forcing leaves exposure bias between training and generation
- − parameters per layer, a third more than an encoder layer, because of the cross-attention block
Check yourself
In a trained encoder–decoder Transformer, which attention matrices must be recomputed at every generation step if keys and values are cached?
Check yourself
During training a decoder reaches near-zero loss within a few hundred steps, but generates garbage. What is the most likely bug?
Remember
- ✓Decoder layer: ; parameters (4,204,032 at ).
- ✓Autoregressive: ; the causal mask makes row depend only on positions .
- ✓Training: shifted-right gold targets, all positions in one pass (teacher forcing); inference: one token per step.
- ✓KV cache: store per layer ( values per token) and the cross-attention of the memory once; instead of query–key products.
- ✓Original decoding: beam 4, length penalty α = 0.6, max length input + 50; label smoothing 0.1 and shared input/output embeddings.