Transformer decoder

A Transformer decoder layer stacks masked self-attention over the outputs so far, cross-attention to the encoder's output and a feed-forward network. It is trained on all positions in parallel with teacher forcing and generates autoregressively, one token per step, reusing cached keys and values.

14 min read1088 wordscore0 viewsupdated 2026-10-12

Definition

Transformer decoder is the half of the Transformer that generates the output sequence. It models p(y∣x)=∏ip(yi∣y<i,x)p(y\mid x)=\prod_i p(y_i\mid y_{<i},x). In the original encoder–decoder model each of its NN identical layers has three sublayers: masked multi-head self-attention over the target positions, in which position ii may only look at positions ≤i\le i; cross-attention whose queries come from the decoder and whose keys and values come from the encoder output; and a position-wise feed-forward network. Each sublayer is wrapped in a residual connection and LayerNorm. A final linear layer and softmax give the distribution of the next token.

The decoder was introduced with the encoder in Vaswani et al., "Attention Is All You Need" (NeurIPS 2017; see Encoder–decoder transformer). It replaced the recurrent decoders of Bahdanau attention and Luong attention, where attention was an add-on to an LSTM. Here the decoder itself is built from attention. Two properties had to be kept from the RNN decoders. The model must be autoregressive: the prediction for yiy_i may depend only on xx and y<iy_{<i}. It must also stay trainable in parallel. The causal mask does both: all target positions are computed in one pass, and none of them can see the future.

Remove the cross-attention sublayer and the decoder becomes a language model of yy alone. That is the Decoder-only transformer of GPT and its successors. This article covers the decoder as part of the encoder–decoder model, where it is conditioned on a source sequence.

One layer

a=LN(z+MHA(z,z,z; causal mask))c=LN(a+MHA(a, m, m; source pad mask))z′=LN(c+FFN(c))p(yi∣y<i,x)=softmax(zi(N) E⊤)\begin{aligned}a&=\mathrm{LN}\big(z+\mathrm{MHA}(z,z,z;\ \text{causal mask})\big)\\ c&=\mathrm{LN}\big(a+\mathrm{MHA}(a,\,m,\,m;\ \text{source pad mask})\big)\\ z'&=\mathrm{LN}\big(c+\mathrm{FFN}(c)\big)\end{aligned}\qquad\qquad p(y_i\mid y_{<i},x)=\mathrm{softmax}\big(z^{(N)}_i\,E^\top\big)
z∈RT×dz\in\mathbb R^{T\times d}
layer input; for layer 1 the embeddings of the shifted-right targets ⟨s⟩,y1,…,yT−1\langle s\rangle,y_1,\dots,y_{T-1}, times d\sqrt d, plus positional encodings
causal mask\text{causal mask}
score (i,j)(i,j) set to −∞-\infty for j>ij>i, so row ii is a softmax over positions 0…i0\dots i only (see Causal attention)
m∈RS×dm\in\mathbb R^{S\times d}
the encoder's final output (the "memory"); the same mm is read by every decoder layer
MHA(a,m,m)\mathrm{MHA}(a,m,m)
Cross-attention: queries aWQaW_Q, keys mWKmW_K, values mWVmW_V; a T×ST\times S weight matrix with no causal mask
E∈RV×dE\in\mathbb R^{V\times d}
the token embedding matrix; the 2017 model shares it between both embedding layers and the output projection
LN\mathrm{LN}
post-LN as in the original; pre-LN decoders apply LN to each sublayer input and once after the stack (see Transformer encoder)

The order matters. Masked self-attention first collects what has been generated so far into each position. Cross-attention then uses that summary as a query, asking which source tokens are relevant for the next output. The FFN processes the result per position. Compared with an encoder layer, a decoder layer has one more attention block: 16d2+19d16d^2+19d parameters instead of 12d2+13d12d^2+13d. For d=512d=512 that is 4,204,032 against 3,152,384.

Training: all positions at once

With teacher forcing (see Teacher forcing) the decoder input is the gold target shifted right by one, and the training target at position ii is yi+1y_{i+1}. All TT positions are computed in one forward pass, with one T×TT\times T masked self-attention per head and one T×ST\times S cross-attention. The loss is the sum of TT cross-entropies. An RNN decoder needs TT sequential steps for the same work.

L=−∑i=1Tlog⁡pθ(yi∣y<i,x),input=(⟨s⟩,y1,…,yT−1),target=(y1,…,yT)\mathcal L=-\sum_{i=1}^{T}\log p_\theta\big(y_i\mid y_{<i},x\big),\qquad \text{input}=(\langle s\rangle,y_1,\dots,y_{T-1}),\qquad \text{target}=(y_1,\dots,y_T)
⟨s⟩\langle s\rangle
start token; the last target is the end token ⟨/s⟩
pθ(yi∣⋅)p_\theta(y_i\mid\cdot)
row i−1i-1 of the output softmax; because of the mask it cannot depend on yi,…,yTy_i,\dots,y_T

The mask is what makes the shift honest. Decoder input position i+1i+1 is the target of position ii. Without the mask, row ii can attend to it and copy the answer. The training loss then drops towards zero, and the model is useless at inference, where that token does not exist yet. Vaswani et al. also used label smoothing with εls=0.1\varepsilon_{ls}=0.1, which in their words "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score".

Inference: one token per step

At test time there is no gold target. The encoder runs once. The decoder starts from ⟨s⟩\langle s\rangle, computes p(⋅∣⟨s⟩,x)p(\cdot\mid\langle s\rangle,x), picks a token (greedy, sampling or Beam search), appends it and repeats until ⟨/s⟩. Done naively, step tt re-runs the decoder on the whole prefix of length tt. Because of the causal mask, the states of positions <t<t are the same as at the previous step, so they can be stored instead of recomputed:

K≤t(ℓ)=[K<t(ℓ); zt(ℓ)WK],V≤t(ℓ)=[V<t(ℓ); zt(ℓ)WV],at(ℓ)=softmax(qt(ℓ)K≤t(ℓ)⊤dk)V≤t(ℓ)K^{(\ell)}_{\le t}=\big[K^{(\ell)}_{<t};\ z^{(\ell)}_tW_K\big],\qquad V^{(\ell)}_{\le t}=\big[V^{(\ell)}_{<t};\ z^{(\ell)}_tW_V\big],\qquad a^{(\ell)}_t=\mathrm{softmax}\Big(\frac{q^{(\ell)}_t K^{(\ell)\top}_{\le t}}{\sqrt{d_k}}\Big)V^{(\ell)}_{\le t}
K<t(ℓ),V<t(ℓ)K^{(\ell)}_{<t},V^{(\ell)}_{<t}
the KV cache of layer ℓ\ell: keys and values of all earlier positions, 2 t d2\,t\,d numbers per layer
qt(ℓ)q^{(\ell)}_t
only the newest position needs a query; every step processes a single token per layer
mWK, mWVmW_K,\ mW_V
the cross-attention keys and values depend only on the encoder output; they are computed once per layer and reused at every step

With the cache, generating TT tokens costs TT new key/value projections and T(T+1)/2T(T+1)/2 query–key products per layer and head. Re-running the prefix costs T(T+1)/2T(T+1)/2 projections and ∑tt2=T(T+1)(2T+1)/6\sum_t t^2=T(T+1)(2T+1)/6 products. For T=100T=100 that is 5,050 against 338,350 products. The price is memory. The cache holds 2Ld2Ld numbers per generated token, plus the fixed 2LdS2LdS of the cross-attention memory, and it is multiplied by the batch size and the beam width.

The decoder in Vaswani et al. (2017)

Stack
N=6N=6 layers; base d=512d=512, dff=2048d_{ff}=2048, 8 heads; big d=1024d=1024, dff=4096d_{ff}=4096, 16 heads
Sublayers
masked self-attention, encoder–decoder attention, FFN; each LN(x + Dropout(Sublayer(x)))
Embeddings
one matrix shared by the two embedding layers and the pre-softmax projection; embeddings multiplied by dmodel\sqrt{d_{\text{model}}}
Regularisation
residual dropout Pdrop=0.1P_{drop}=0.1 (base), label smoothing εls=0.1\varepsilon_{ls}=0.1
Decoding
beam search with beam size 4 and length penalty α=0.6\alpha=0.6; maximum output length = input length + 50, stopping early when possible
Parameters per layer
16d2+19d16d^2+19d: 4,204,032 (base), 16,796,672 (big)
inference: step 4 of 9: query = input 3 (noir), predicts output 4masked self-attn<s>lechatnoirdortsurlevieuxcanapé<s>lechatnoir·····causal: keys j ≤ i onlycross-attn → encoder memorytheblackcatsleepsontheoldsofa</s>→ le→ chat→ noir→ dortcols: source; no maskposition 3 · in noirtarget dortp(target) 0.767cross max sleeps 0.89argmax ok 4/4mask causalw on j > i 0.000mean, 9 rows 0.000softmax at position 3le0.00chat0.00noir0.21dort0.77sur0.00vieux0.00canapé0.00</s>0.01la0.00chien0.00per-layer work, steps 1…4 (KV cache)self k, v vectors 4 (cache 4 / none 10)cross k, v vectors 9 (9 / 36)self q·k 10 (10 / 30)cross q·k 36 (36 / 90)base model (6 layers, d=512) self cacheafter 4 tokens: 24,576 values = 48 KiB fp16+ cross k, v of the memory: 55,296 (fixed)source: the black cat sleeps on the old sofa </s>output: le chat noir dort _ _ _ _ _
phase
causal maskKV cache

Mask-off is a training-only bug: at inference the future does not exist yet.

fig.One decoder layer translating 'the black cat sleeps on the old sofa' into 'le chat noir dort sur le vieux canapé'. The vectors are fixed random stand-ins, not a trained model: each cross-attention query is a noisy copy of the source word to translate. Inference, step 4: the query at input 'noir' attends to positions 0–3 only, cross-attention puts 0.89 on 'sleeps', and the softmax gives 'dort' 0.767. All 9 greedy tokens come out right, but the first 'le' splits its cross-attention 0.29 / 0.56 between the two copies of 'the'. Content alone cannot separate them, which is why positional information matters. Switch to training: all 9 rows are computed in one pass, 81 query–key products of which 45 survive the mask. Turn the mask off and row 2 ('chat', which must predict 'noir') puts 0.856 of its weight on later positions, which include the answer itself; the mean over rows is 0.494. The work counters for 9 steps, per layer: with the KV cache 9 self K/V vectors, 9 cross K/V vectors, 45 self q·k and 81 cross q·k; re-running the prefix needs 45, 81, 285 and 405. A 6-layer d = 512 decoder caches 55,296 values (108 KiB in fp16) after 9 tokens.

Worked example: a masked row and the cost of decoding

Decoder position 2 (query) has scaled scores (1.0, 0.2, 2.0)(1.0,\ 0.2,\ 2.0) against keys 0, 1, 2 and 3.03.0 against key 3, a future position. Values v0=(1,0)v_0=(1,0), v1=(0,1)v_1=(0,1), v2=(1,1)v_2=(1,1).

key jscoreexpweight, causalweight, no mask
01.02.71830.23990.0865
10.21.22140.10780.0389
22.07.38910.65220.2352
3 (future)3.020.08550 (masked)0.6394
Masked vs unmasked softmax of row 2 (computed in Python)

From weights to output, then to cost

  1. 1Causal: the normaliser is 2.7183+1.2214+7.3891=11.32872.7183+1.2214+7.3891=11.3287, and the output is 0.2399 v0+0.1078 v1+0.6522 v2=(0.8922, 0.7601)0.2399\,v_0+0.1078\,v_1+0.6522\,v_2=(0.8922,\ 0.7601).
  2. 2Without the mask the normaliser is 31.414331.4143 and 64% of the weight goes to position 3. In teacher forcing that is the input token y3y_3, the very token position 2 is trained to predict.
  3. 3Cost of generating T=100T=100 tokens, per layer and head: with a KV cache, 100 key/value projections and ∑t=1100t=5,050\sum_{t=1}^{100}t=5{,}050 products; re-running the prefix, 5,0505{,}050 projections and ∑t2=338,350\sum t^2=338{,}350 products, 67 times as many.
  4. 4Memory for Transformer big (L=6L=6, d=1024d=1024, fp16): 2⋅6⋅1024=12,2882\cdot6\cdot1024=12{,}288 cached values, 24 KiB, per token. 100 tokens take 2.34 MiB per sequence, and 9.38 MiB for a beam of 4, plus the cross-attention keys and values of the source.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class DecoderLayer(nn.Module):
4 def __init__(self, d=512, h=8, d_ff=2048, p=0.1):
5 super().__init__()
6 self.h = h
7 self.qkv, self.o1 = nn.Linear(d, 3 * d), nn.Linear(d, d) # masked self-attention
8 self.q2, self.kv2, self.o2 = nn.Linear(d, d), nn.Linear(d, 2 * d), nn.Linear(d, d) # cross
9 self.ff = nn.Sequential(nn.Linear(d, d_ff), nn.ReLU(), nn.Dropout(p), nn.Linear(d_ff, d))
10 self.n1, self.n2, self.n3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d)
11 self.drop = nn.Dropout(p)
12
13 def split(self, x): # (N, L, d) -> (N, h, L, d/h)
14 return x.unflatten(-1, (self.h, -1)).transpose(1, 2)
15
16 def merge(self, x): # (N, h, L, d/h) -> (N, L, d)
17 return x.transpose(1, 2).flatten(-2)
18
19 def forward(self, y, mem, mem_pad=None, cache=None):
20 # y (N, T_new, d): new decoder positions; mem (N, S, d): encoder output
21 q, k, v = map(self.split, self.qkv(y).chunk(3, -1))
22 if cache is not None:
23 if "k" in cache: # append to the cached keys/values
24 k, v = torch.cat([cache["k"], k], 2), torch.cat([cache["v"], v], 2)
25 cache["k"], cache["v"] = k, v
26 t_new, t_all = q.size(2), k.size(2)
27 keep = torch.ones(t_new, t_all, dtype=torch.bool, device=y.device).tril(t_all - t_new)
28 a = F.scaled_dot_product_attention(q, k, v, attn_mask=keep) # True = may attend
29 y = self.n1(y + self.drop(self.o1(self.merge(a))))
30
31 if cache is not None and "mk" in cache: # cross K, V: computed once per sequence
32 mk, mv = cache["mk"], cache["mv"]
33 else:
34 mk, mv = map(self.split, self.kv2(mem).chunk(2, -1))
35 if cache is not None:
36 cache["mk"], cache["mv"] = mk, mv
37 keep_src = None if mem_pad is None else ~mem_pad[:, None, None, :]
38 a = F.scaled_dot_product_attention(self.split(self.q2(y)), mk, mv, attn_mask=keep_src)
39 y = self.n2(y + self.drop(self.o2(self.merge(a))))
40 return self.n3(y + self.drop(self.ff(y)))
41
42torch.manual_seed(0)
43layer = DecoderLayer(d=64, h=4, d_ff=256).eval()
44mem, y = torch.randn(2, 7, 64), torch.randn(2, 5, 64)
45
46full = layer(y, mem) # training-style: all 5 positions at once
47cache = {}
48steps = [layer(y[:, t:t + 1], mem, cache=cache) for t in range(5)] # inference-style
49print(full.shape, torch.allclose(full, torch.cat(steps, 1), atol=1e-5)) # torch.Size([2, 5, 64]) True
50print(cache["k"].shape, cache["mk"].shape) # torch.Size([2, 4, 5, 16]) torch.Size([2, 4, 7, 16])
51print(sum(p.numel() for p in DecoderLayer().parameters())) # 4204032 = 16*512**2 + 19*512
52print(sum(p.numel() for p in nn.TransformerDecoderLayer(512, 8, 2048).parameters())) # 4204032
A decoder layer with self- and cross-attention and a KV cache; incremental decoding matches the parallel pass

Use when

  • +Training is parallel over all target positions: one masked pass instead of T recurrent steps
  • +Every layer cross-attends to the full encoder memory, so no fixed-size bottleneck vector
  • +Exact autoregressive factorisation: likelihoods, sampling and beam search all work directly
  • +A KV cache makes each generation step cost O(t·d) in attention instead of re-running the prefix

Avoid when

  • −Generation is still sequential: T tokens need T decoder passes
  • −The KV cache grows as 2Ld2Ld values per token per sequence, times batch and beam
  • −Teacher forcing leaves exposure bias between training and generation
  • −16d216d^2 parameters per layer, a third more than an encoder layer, because of the cross-attention block

Check yourself

In a trained encoder–decoder Transformer, which attention matrices must be recomputed at every generation step if keys and values are cached?

Check yourself

During training a decoder reaches near-zero loss within a few hundred steps, but generates garbage. What is the most likely bug?

Remember

  • ✓Decoder layer: LN(z+MHAcausal(z))→LN(a+MHA(a,m,m))→LN(c+FFN(c))\mathrm{LN}(z+\mathrm{MHA}_{\text{causal}}(z))\to\mathrm{LN}(a+\mathrm{MHA}(a,m,m))\to\mathrm{LN}(c+\mathrm{FFN}(c)); 16d2+19d16d^2+19d parameters (4,204,032 at d=512d=512).
  • ✓Autoregressive: p(y∣x)=∏ip(yi∣y<i,x)p(y\mid x)=\prod_i p(y_i\mid y_{<i},x); the causal mask makes row ii depend only on positions ≤i\le i.
  • ✓Training: shifted-right gold targets, all positions in one pass (teacher forcing); inference: one token per step.
  • ✓KV cache: store K,VK,V per layer (2Ld2Ld values per token) and the cross-attention K,VK,V of the memory once; T(T+1)/2T(T+1)/2 instead of T(T+1)(2T+1)/6T(T+1)(2T+1)/6 query–key products.
  • ✓Original decoding: beam 4, length penalty α = 0.6, max length input + 50; label smoothing 0.1 and shared input/output embeddings.
← back to Deep Learning roadmap