# Transformer decoder

> A Transformer decoder layer stacks masked self-attention over the outputs so far, cross-attention to the encoder's output and a feed-forward network. It is trained on all positions in parallel with teacher forcing and generates autoregressively, one token per step, reusing cached keys and values.

Source: https://www.mlfoundry.in/learn/transformer-decoder  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 14 min read · updated 2026-10-12

**Transformer decoder:** is the half of the Transformer that generates the output sequence. It models $p(y\mid x)=\prod_i p(y_i\mid y_{<i},x)$. In the original encoder–decoder model each of its $N$ identical layers has three sublayers: **masked multi-head self-attention** over the target positions, in which position $i$ may only look at positions $\le i$; **cross-attention** whose queries come from the decoder and whose keys and values come from the encoder output; and a position-wise **feed-forward network**. Each sublayer is wrapped in a residual connection and LayerNorm. A final linear layer and softmax give the distribution of the next token.

The decoder was introduced with the encoder in Vaswani et al., "Attention Is All You Need" (NeurIPS 2017; see **Encoder–decoder transformer**). It replaced the recurrent decoders of **Bahdanau attention** and **Luong attention**, where attention was an add-on to an LSTM. Here the decoder itself is built from attention. Two properties had to be kept from the RNN decoders. The model must be **autoregressive**: the prediction for $y_i$ may depend only on $x$ and $y_{<i}$. It must also stay trainable in parallel. The causal mask does both: all target positions are computed in one pass, and none of them can see the future.

Remove the cross-attention sublayer and the decoder becomes a language model of $y$ alone. That is the **Decoder-only transformer** of GPT and its successors. This article covers the decoder as part of the encoder–decoder model, where it is conditioned on a source sequence.

## One layer

$$
\begin{aligned}a&=\mathrm{LN}\big(z+\mathrm{MHA}(z,z,z;\ \text{causal mask})\big)\\ c&=\mathrm{LN}\big(a+\mathrm{MHA}(a,\,m,\,m;\ \text{source pad mask})\big)\\ z'&=\mathrm{LN}\big(c+\mathrm{FFN}(c)\big)\end{aligned}\qquad\qquad p(y_i\mid y_{<i},x)=\mathrm{softmax}\big(z^{(N)}_i\,E^\top\big)
$$

- $z\in\mathbb R^{T\times d}$: layer input; for layer 1 the embeddings of the **shifted-right** targets $\langle s\rangle,y_1,\dots,y_{T-1}$, times $\sqrt d$, plus positional encodings
- $\text{causal mask}$: score $(i,j)$ set to $-\infty$ for $j>i$, so row $i$ is a softmax over positions $0\dots i$ only (see **Causal attention**)
- $m\in\mathbb R^{S\times d}$: the encoder's final output (the "memory"); the **same** $m$ is read by every decoder layer
- $\mathrm{MHA}(a,m,m)$: **Cross-attention**: queries $aW_Q$, keys $mW_K$, values $mW_V$; a $T\times S$ weight matrix with no causal mask
- $E\in\mathbb R^{V\times d}$: the token embedding matrix; the 2017 model shares it between both embedding layers and the output projection
- $\mathrm{LN}$: post-LN as in the original; pre-LN decoders apply LN to each sublayer input and once after the stack (see **Transformer encoder**)

The order matters. Masked self-attention first collects what has been generated so far into each position. Cross-attention then uses that summary as a query, asking which source tokens are relevant for the next output. The FFN processes the result per position. Compared with an encoder layer, a decoder layer has one more attention block: $16d^2+19d$ parameters instead of $12d^2+13d$. For $d=512$ that is 4,204,032 against 3,152,384.

## Training: all positions at once

With **teacher forcing** (see **Teacher forcing**) the decoder input is the gold target shifted right by one, and the training target at position $i$ is $y_{i+1}$. All $T$ positions are computed in one forward pass, with one $T\times T$ masked self-attention per head and one $T\times S$ cross-attention. The loss is the sum of $T$ cross-entropies. An RNN decoder needs $T$ sequential steps for the same work.

$$
\mathcal L=-\sum_{i=1}^{T}\log p_\theta\big(y_i\mid y_{<i},x\big),\qquad \text{input}=(\langle s\rangle,y_1,\dots,y_{T-1}),\qquad \text{target}=(y_1,\dots,y_T)
$$

- $\langle s\rangle$: start token; the last target is the end token ⟨/s⟩
- $p_\theta(y_i\mid\cdot)$: row $i-1$ of the output softmax; because of the mask it cannot depend on $y_i,\dots,y_T$

The mask is what makes the shift honest. Decoder input position $i+1$ *is* the target of position $i$. Without the mask, row $i$ can attend to it and copy the answer. The training loss then drops towards zero, and the model is useless at inference, where that token does not exist yet. Vaswani et al. also used **label smoothing** with $\varepsilon_{ls}=0.1$, which in their words "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score".

## Inference: one token per step

At test time there is no gold target. The encoder runs once. The decoder starts from $\langle s\rangle$, computes $p(\cdot\mid\langle s\rangle,x)$, picks a token (greedy, sampling or **Beam search**), appends it and repeats until ⟨/s⟩. Done naively, step $t$ re-runs the decoder on the whole prefix of length $t$. Because of the causal mask, the states of positions $<t$ are the same as at the previous step, so they can be stored instead of recomputed:

$$
K^{(\ell)}_{\le t}=\big[K^{(\ell)}_{<t};\ z^{(\ell)}_tW_K\big],\qquad V^{(\ell)}_{\le t}=\big[V^{(\ell)}_{<t};\ z^{(\ell)}_tW_V\big],\qquad a^{(\ell)}_t=\mathrm{softmax}\Big(\frac{q^{(\ell)}_t K^{(\ell)\top}_{\le t}}{\sqrt{d_k}}\Big)V^{(\ell)}_{\le t}
$$

- $K^{(\ell)}_{<t},V^{(\ell)}_{<t}$: the **KV cache** of layer $\ell$: keys and values of all earlier positions, $2\,t\,d$ numbers per layer
- $q^{(\ell)}_t$: only the newest position needs a query; every step processes a single token per layer
- $mW_K,\ mW_V$: the cross-attention keys and values depend only on the encoder output; they are computed once per layer and reused at every step

With the cache, generating $T$ tokens costs $T$ new key/value projections and $T(T+1)/2$ query–key products per layer and head. Re-running the prefix costs $T(T+1)/2$ projections and $\sum_t t^2=T(T+1)(2T+1)/6$ products. For $T=100$ that is 5,050 against 338,350 products. The price is memory. The cache holds $2Ld$ numbers per generated token, plus the fixed $2LdS$ of the cross-attention memory, and it is multiplied by the batch size and the beam width.

**The decoder in Vaswani et al. (2017)**

- **Stack:** $N=6$ layers; base $d=512$, $d_{ff}=2048$, 8 heads; big $d=1024$, $d_{ff}=4096$, 16 heads
- **Sublayers:** masked self-attention, encoder–decoder attention, FFN; each LN(x + Dropout(Sublayer(x)))
- **Embeddings:** one matrix shared by the two embedding layers and the pre-softmax projection; embeddings multiplied by $\sqrt{d_{\text{model}}}$
- **Regularisation:** residual dropout $P_{drop}=0.1$ (base), label smoothing $\varepsilon_{ls}=0.1$
- **Decoding:** beam search with beam size 4 and length penalty $\alpha=0.6$; maximum output length = input length + 50, stopping early when possible
- **Parameters per layer:** $16d^2+19d$: 4,204,032 (base), 16,796,672 (big)

> Figure: One decoder layer translating 'the black cat sleeps on the old sofa' into 'le chat noir dort sur le vieux canapé'. The vectors are fixed random stand-ins, not a trained model: each cross-attention query is a noisy copy of the source word to translate. Inference, step 4: the query at input 'noir' attends to positions 0–3 only, cross-attention puts 0.89 on 'sleeps', and the softmax gives 'dort' 0.767. All 9 greedy tokens come out right, but the first 'le' splits its cross-attention 0.29 / 0.56 between the two copies of 'the'. Content alone cannot separate them, which is why positional information matters. Switch to training: all 9 rows are computed in one pass, 81 query–key products of which 45 survive the mask. Turn the mask off and row 2 ('chat', which must predict 'noir') puts 0.856 of its weight on later positions, which include the answer itself; the mean over rows is 0.494. The work counters for 9 steps, per layer: with the KV cache 9 self K/V vectors, 9 cross K/V vectors, 45 self q·k and 81 cross q·k; re-running the prefix needs 45, 81, 285 and 405. A 6-layer d = 512 decoder caches 55,296 values (108 KiB in fp16) after 9 tokens.

## Worked example: a masked row and the cost of decoding

Decoder position 2 (query) has scaled scores $(1.0,\ 0.2,\ 2.0)$ against keys 0, 1, 2 and $3.0$ against key 3, a future position. Values $v_0=(1,0)$, $v_1=(0,1)$, $v_2=(1,1)$.

Masked vs unmasked softmax of row 2 (computed in Python)

| key j | score | exp | weight, causal | weight, no mask |
| --- | --- | --- | --- | --- |
| 0 | 1.0 | 2.7183 | 0.2399 | 0.0865 |
| 1 | 0.2 | 1.2214 | 0.1078 | 0.0389 |
| 2 | 2.0 | 7.3891 | 0.6522 | 0.2352 |
| 3 (future) | 3.0 | 20.0855 | 0 (masked) | 0.6394 |

**From weights to output, then to cost**

1. Causal: the normaliser is $2.7183+1.2214+7.3891=11.3287$, and the output is $0.2399\,v_0+0.1078\,v_1+0.6522\,v_2=(0.8922,\ 0.7601)$.
2. Without the mask the normaliser is $31.4143$ and 64% of the weight goes to position 3. In teacher forcing that is the input token $y_3$, the very token position 2 is trained to predict.
3. Cost of generating $T=100$ tokens, per layer and head: with a KV cache, 100 key/value projections and $\sum_{t=1}^{100}t=5{,}050$ products; re-running the prefix, $5{,}050$ projections and $\sum t^2=338{,}350$ products, 67 times as many.
4. Memory for Transformer big ($L=6$, $d=1024$, fp16): $2\cdot6\cdot1024=12{,}288$ cached values, 24 KiB, per token. 100 tokens take 2.34 MiB per sequence, and 9.38 MiB for a beam of 4, plus the cross-attention keys and values of the source.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class DecoderLayer(nn.Module):
    def __init__(self, d=512, h=8, d_ff=2048, p=0.1):
        super().__init__()
        self.h = h
        self.qkv, self.o1 = nn.Linear(d, 3 * d), nn.Linear(d, d)          # masked self-attention
        self.q2, self.kv2, self.o2 = nn.Linear(d, d), nn.Linear(d, 2 * d), nn.Linear(d, d)  # cross
        self.ff = nn.Sequential(nn.Linear(d, d_ff), nn.ReLU(), nn.Dropout(p), nn.Linear(d_ff, d))
        self.n1, self.n2, self.n3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d)
        self.drop = nn.Dropout(p)

    def split(self, x):                            # (N, L, d) -> (N, h, L, d/h)
        return x.unflatten(-1, (self.h, -1)).transpose(1, 2)

    def merge(self, x):                            # (N, h, L, d/h) -> (N, L, d)
        return x.transpose(1, 2).flatten(-2)

    def forward(self, y, mem, mem_pad=None, cache=None):
        # y (N, T_new, d): new decoder positions; mem (N, S, d): encoder output
        q, k, v = map(self.split, self.qkv(y).chunk(3, -1))
        if cache is not None:
            if "k" in cache:                       # append to the cached keys/values
                k, v = torch.cat([cache["k"], k], 2), torch.cat([cache["v"], v], 2)
            cache["k"], cache["v"] = k, v
        t_new, t_all = q.size(2), k.size(2)
        keep = torch.ones(t_new, t_all, dtype=torch.bool, device=y.device).tril(t_all - t_new)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=keep)        # True = may attend
        y = self.n1(y + self.drop(self.o1(self.merge(a))))

        if cache is not None and "mk" in cache:    # cross K, V: computed once per sequence
            mk, mv = cache["mk"], cache["mv"]
        else:
            mk, mv = map(self.split, self.kv2(mem).chunk(2, -1))
            if cache is not None:
                cache["mk"], cache["mv"] = mk, mv
        keep_src = None if mem_pad is None else ~mem_pad[:, None, None, :]
        a = F.scaled_dot_product_attention(self.split(self.q2(y)), mk, mv, attn_mask=keep_src)
        y = self.n2(y + self.drop(self.o2(self.merge(a))))
        return self.n3(y + self.drop(self.ff(y)))

torch.manual_seed(0)
layer = DecoderLayer(d=64, h=4, d_ff=256).eval()
mem, y = torch.randn(2, 7, 64), torch.randn(2, 5, 64)

full = layer(y, mem)                                    # training-style: all 5 positions at once
cache = {}
steps = [layer(y[:, t:t + 1], mem, cache=cache) for t in range(5)]  # inference-style
print(full.shape, torch.allclose(full, torch.cat(steps, 1), atol=1e-5))  # torch.Size([2, 5, 64]) True
print(cache["k"].shape, cache["mk"].shape)              # torch.Size([2, 4, 5, 16]) torch.Size([2, 4, 7, 16])
print(sum(p.numel() for p in DecoderLayer().parameters()))                    # 4204032 = 16*512**2 + 19*512
print(sum(p.numel() for p in nn.TransformerDecoderLayer(512, 8, 2048).parameters()))  # 4204032
```

A decoder layer with self- and cross-attention and a KV cache; incremental decoding matches the parallel pass

> **Shift and mask must agree:** The decoder input is the target shifted right, so input position $i+1$ holds the answer for position $i$. Forget the causal mask, or build it with the wrong diagonal offset (`triu(..., 0)` instead of `triu(..., 1)` for a "masked = True" mask, which also hides the token itself), and training looks excellent while generation fails. A quick test is the one in the code: run the layer on the whole sequence and step by step, and check that the outputs match.

> **Padding on both sides:** A decoder needs up to three masks: the causal mask, the **source** padding mask for cross-attention (so no query reads encoder states of [PAD] tokens), and a target padding mask, or simply `ignore_index` in the loss, so padded target positions do not contribute. The source mask is easy to forget because the decoder never sees source tokens directly, only the memory $m$.

> **Why decoding is memory-bound:** With a KV cache each step processes one token per layer. The matrix multiplies shrink to matrix–vector products, and the step time is set mostly by reading the weights and the growing cache from memory, not by arithmetic. This is the motivation for **Multi-query attention (MQA)** and **Grouped-query attention (GQA)**, which share keys and values across heads to shrink the cache, and for kernels such as **FlashAttention**.

> **Exposure bias:** Teacher forcing trains every position on gold prefixes, but generation conditions on the model's own outputs. An early mistake gives the decoder a prefix it never saw in training. This is the same train–test mismatch as in RNN decoders (see **Scheduled sampling**). Beam search and label smoothing reduce its effect. Neither removes it.

**Strengths**

- Training is parallel over all target positions: one masked pass instead of T recurrent steps
- Every layer cross-attends to the full encoder memory, so no fixed-size bottleneck vector
- Exact autoregressive factorisation: likelihoods, sampling and beam search all work directly
- A KV cache makes each generation step cost O(t·d) in attention instead of re-running the prefix

**Limitations**

- Generation is still sequential: T tokens need T decoder passes
- The KV cache grows as $2Ld$ values per token per sequence, times batch and beam
- Teacher forcing leaves exposure bias between training and generation
- $16d^2$ parameters per layer, a third more than an encoder layer, because of the cross-attention block

**Question:** In a trained encoder–decoder Transformer, which attention matrices must be recomputed at every generation step if keys and values are cached?

- A. All of them, including the encoder self-attention
- B. Only the new query row of each decoder self-attention and each cross-attention
- C. Only the encoder self-attention
- D. None; the whole output is computed once

**Answer:** B. Only the new query row of each decoder self-attention and each cross-attention — The encoder runs once, and its output gives the cross-attention keys and values once per layer. In decoder self-attention, earlier positions cannot see the new token, so their states do not change and are cached. Each step therefore computes one query row: $1\times t$ in self-attention and $1\times S$ in cross-attention, per layer and head.

**Question:** During training a decoder reaches near-zero loss within a few hundred steps, but generates garbage. What is the most likely bug?

- A. The learning rate is too low
- B. The causal mask is missing or off by one, so each position can read the next input token, which is its own target
- C. Label smoothing is enabled
- D. The source padding mask is missing

**Answer:** B. The causal mask is missing or off by one, so each position can read the next input token, which is its own target — With inputs shifted right, input $i+1$ equals target $i$. An unmasked or wrongly offset mask lets row $i$ copy it, so training is trivial. At inference that token is not available yet, and the model has learned nothing useful. In the worked example 64% of the unmasked weight lands on the future position. A missing source mask hurts quality but does not cause near-zero loss.

**Key points**

- Decoder layer: $\mathrm{LN}(z+\mathrm{MHA}_{\text{causal}}(z))\to\mathrm{LN}(a+\mathrm{MHA}(a,m,m))\to\mathrm{LN}(c+\mathrm{FFN}(c))$; $16d^2+19d$ parameters (4,204,032 at $d=512$).
- Autoregressive: $p(y\mid x)=\prod_i p(y_i\mid y_{<i},x)$; the causal mask makes row $i$ depend only on positions $\le i$.
- Training: shifted-right gold targets, all positions in one pass (teacher forcing); inference: one token per step.
- KV cache: store $K,V$ per layer ($2Ld$ values per token) and the cross-attention $K,V$ of the memory once; $T(T+1)/2$ instead of $T(T+1)(2T+1)/6$ query–key products.
- Original decoding: beam 4, length penalty α = 0.6, max length input + 50; label smoothing 0.1 and shared input/output embeddings.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
