# Encoder–decoder transformer

> The encoder–decoder Transformer (Vaswani et al., 2017) reads the source with a bidirectional self-attention stack and generates the target with a causal decoder that cross-attends to the encoder output in every layer. It set WMT'14 records (28.4 BLEU En–De) and underlies T5 and BART.

Source: https://www.mlfoundry.in/learn/encoder-decoder-transformer  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 16 min read · updated 2026-10-12

**Encoder–decoder transformer:** is a sequence-to-sequence model built only from attention and feed-forward layers. An **encoder** of $N$ layers turns the source $x_1,\dots,x_S$ into a memory $m\in\mathbb R^{S\times d}$ using unmasked self-attention. A **decoder** of $N$ layers generates $y_1,\dots,y_T$ one token at a time using causal self-attention over its own prefix, **cross-attention** to $m$ in every layer, and a softmax over the vocabulary. It models $p(y\mid x)=\prod_{i}p(y_i\mid y_{<i},m)$ and is trained end to end with cross-entropy.

The model is the Transformer of Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, "Attention Is All You Need" (NeurIPS 2017). In 2017 the best translation systems were recurrent or convolutional encoder–decoders with attention (see **Sequence-to-sequence (Seq2Seq)** and **Bahdanau attention**). Recurrence forces sequential computation over the positions of a sentence, so the work cannot be parallelised within a training example. The paper kept the encoder–decoder structure and the attention between the two halves, and replaced every recurrent and convolutional layer with self-attention. A single big model trained for 3.5 days on eight P100 GPUs set new records on WMT'14 English→German and English→French at a fraction of the training cost of the previous best models.

The same architecture was later pre-trained as a general text-to-text model, notably in **T5** (Raffel et al., JMLR 2020) and **BART** (Lewis et al., ACL 2020). It remains the standard choice when the input and output are distinct sequences: translation, summarisation, speech recognition (Whisper), and many structured prediction tasks.

## The architecture

$$
\begin{aligned}e_0&=\sqrt d\,E[x]+P_{1:S},\qquad m=\mathrm{Enc}_N\circ\dots\circ\mathrm{Enc}_1(e_0)\\ z_0&=\sqrt d\,E[\langle s\rangle,y_1,\dots,y_{T-1}]+P_{1:T},\qquad z_N=\mathrm{Dec}_N(\dots\mathrm{Dec}_1(z_0,m)\dots,m)\\ p(y_i\mid y_{<i},x)&=\mathrm{softmax}\big(z_{N,i}\,E^\top\big)\end{aligned}
$$

- $E\in\mathbb R^{V\times d}$: one embedding matrix shared by the source embedding, the target embedding and the output projection (the paper used a shared source–target vocabulary for this)
- $P$: sinusoidal positional encodings, $P_{p,2k}=\sin(p/10000^{2k/d})$, $P_{p,2k+1}=\cos(p/10000^{2k/d})$ (see **Positional encoding**)
- $\mathrm{Enc}_\ell$: self-attention + FFN with residuals and post-LN (see **Transformer encoder**)
- $\mathrm{Dec}_\ell(z,m)$: masked self-attention, cross-attention to $m$, FFN (see **Transformer decoder**)
- $m$: the encoder output of the **last** layer, read by the cross-attention of **every** decoder layer

Attention appears in three forms, with three mask patterns. In **encoder self-attention** an $S\times S$ matrix is fully visible except for source padding. In **decoder self-attention** a $T\times T$ matrix is lower-triangular. In **cross-attention** a $T\times S$ matrix has queries from the decoder and keys and values from $m$, and it is fully visible except for source padding. Each uses $h$ heads of width $d/h$ (see **Multi-head attention**) and the scaled dot product $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$ (see **Scaled dot-product attention**).

The paper compared costs per layer. A self-attention layer costs $O(n^2d)$ with $O(1)$ sequential operations and maximum path length $O(1)$. A recurrent layer costs $O(nd^2)$ with $O(n)$ sequential operations and path length $O(n)$. For sentences, where $n<d$, self-attention is also cheaper in FLOPs. Every source and target position is connected to every other through a constant number of layers.

Base and big configurations (Vaswani et al. 2017, Table 3)

| model | N | d_model | d_ff | h | d_k = d_v | P_drop | ε_ls | train steps | params |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| base | 6 | 512 | 2048 | 8 | 64 | 0.1 | 0.1 | 100K | 65M |
| big | 6 | 1024 | 4096 | 16 | 64 | 0.3 | 0.1 | 300K | 213M |

**Training setup in the paper**

- **Data:** WMT 2014 En–De: about 4.5M sentence pairs, byte-pair encoding with a shared source–target vocabulary of about 37,000 tokens. En–Fr: 36M sentences, 32,000 word-piece vocabulary
- **Batches:** sentence pairs batched by approximate length; about 25,000 source and 25,000 target tokens per batch
- **Hardware:** one machine with 8 NVIDIA P100 GPUs; base 0.4 s per step (100K steps, 12 hours), big 1.0 s per step (300K steps, 3.5 days)
- **Optimiser:** Adam, $\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-9}$; $\mathrm{lrate}=d_{\text{model}}^{-0.5}\min(\text{step}^{-0.5},\ \text{step}\cdot\text{warmup}^{-1.5})$ with warmup = 4000
- **Regularisation:** dropout on every sublayer output and on the embedding + PE sums; label smoothing $\varepsilon_{ls}=0.1$; the big En–Fr model used $P_{drop}=0.1$
- **Inference:** averaged the last 5 checkpoints (base) or 20 (big); beam size 4, length penalty α = 0.6, max length input + 50

## Results

WMT'14 newstest2014 BLEU and estimated training cost (Vaswani et al. 2017, Table 2, selected rows)

| model | EN-DE | EN-FR | training FLOPs (EN-DE) |
| --- | --- | --- | --- |
| GNMT + RL | 24.6 | 39.92 | 2.3 · 10¹⁹ |
| ConvS2S | 25.16 | 40.46 | 9.6 · 10¹⁸ |
| GNMT + RL ensemble | 26.30 | 41.16 |  |
| ConvS2S ensemble | 26.36 | 41.29 |  |
| Transformer (base) | 27.3 | 38.1 | 3.3 · 10¹⁸ |
| Transformer (big) | 28.4 | 41.8 | 2.3 · 10¹⁹ |

The big model beat the best previous results, ensembles included, by more than 2 BLEU on English→German. On English→German the base model already beat all of them, with $3.3\cdot10^{18}$ training FLOPs against $9.6\cdot10^{18}$ for ConvS2S and $2.3\cdot10^{19}$ for GNMT + RL. The paper computed the FLOP estimates as training time × number of GPUs × an assumed sustained 9.5 TFLOPS per P100. The English→French score of the big model is 41.8 in the revised arXiv version. The NeurIPS proceedings version reports 41.0. The same architecture, with 4 layers and $d=1024$, also reached 91.3 F1 on WSJ constituency parsing with little task-specific tuning.

> Figure: The full model, with the three attention patterns for a source of S = 10 and a target of T = 8 tokens: 100 allowed query–key pairs in encoder self-attention, 36 in causal decoder self-attention and 80 in cross-attention, 216 per head and layer. Click a box or a pattern to see its share of the parameters. These are exact counts with biases and LayerNorms. Transformer base comes to 63,082,496 (paper: 65M; the exact vocabulary size is not given). Of that, 30.0% is the shared 37K × 512 embedding, 20.0% each FFN stack and 10.0% each attention type: an encoder layer has 3,152,384 parameters, a decoder layer 4,204,032. Transformer big gives 214,245,376 (paper: 213M). T5-base, with no biases, scale-only LayerNorms, relative position biases and a 32,128 vocabulary, gives 222,903,552 (paper: 220M). BART-large, with learned positions and a 50,265 vocabulary, gives 406,291,456. Its embedding share falls to 12.7%, and each FFN stack is 24.8%.

## Worked example: counting Transformer base and its schedule

Exact parameter count of Transformer base with a 37,000-token shared vocabulary (computed)

| component | formula | parameters | share |
| --- | --- | --- | --- |
| shared embedding / output | V·d = 37,000 · 512 | 18,944,000 | 30.0% |
| encoder, 6 layers | 6 · (12d² + 13d) | 18,914,304 | 30.0% |
| decoder self-attention + LN | 6 · (4d² + 4d + 2d) | 6,309,888 | 10.0% |
| decoder cross-attention + LN | 6 · (4d² + 4d + 2d) | 6,309,888 | 10.0% |
| decoder FFN + LN | 6 · (8d² + 5d + 2d) | 12,604,416 | 20.0% |
| total |  | 63,082,496 | 100% |

**Parameters, compute and learning rate**

1. Per layer with $d=512$, $d_{ff}=2048$: encoder $12d^2+13d=3{,}152{,}384$, decoder $16d^2+19d=4{,}204{,}032$. Six of each give $44{,}138{,}496$, and the embedding adds $18{,}944{,}000$, for $63.08$M. The paper reports 65M. The difference most likely comes from the exact BPE vocabulary, which is given only as "about 37000".
2. Training cost of base: $12\ \text{h}\times3600\times8\ \text{GPUs}\times9.5\cdot10^{12}=3.28\cdot10^{18}$ FLOPs, the $3.3\cdot10^{18}$ of Table 2. Big: $3.5\times86{,}400\times8\times9.5\cdot10^{12}=2.30\cdot10^{19}$.
3. Learning rate: $\mathrm{lrate}(s)=512^{-0.5}\min(s^{-0.5},\,s\cdot4000^{-1.5})$ rises linearly to its peak at $s=4000$, $0.044194\times0.015811=6.99\cdot10^{-4}$. Then it decays as $s^{-0.5}$: $3.49\cdot10^{-4}$ at $16{,}000$ and $1.40\cdot10^{-4}$ at $100{,}000$.
4. Allowed query–key pairs per head and layer for $S=10$, $T=8$: $S^2=100$ (encoder) $+\ T(T+1)/2=36$ (decoder) $+\ TS=80$ (cross) $=216$.

## T5 and BART: pre-trained encoder–decoders

**T5** (Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li and Liu, "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", JMLR 2020) casts every task as text in, text out: "translate English to German: …", "summarize: …", or a GLUE label written as a word. It pre-trains on C4, a cleaned Common Crawl corpus of about 750 GB, with **span corruption**. 15% of tokens are dropped in spans of mean length 3. Each span is replaced by a sentinel token in the input, and the target lists the dropped spans after their sentinels. In the paper's controlled comparison, the encoder–decoder with this denoising objective beat decoder-only and prefix-LM variants in comparable settings. Architecturally T5 changes the 2017 model in small ways: pre-LN with a scale-only LayerNorm, no bias terms, and relative position biases (32 buckets, a learned scalar per head added to the attention logits) in place of sinusoids. The released sizes were Small (60M), Base (220M), Large (770M), 3B and 11B.

**BART** (Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov and Zettlemoyer, "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension", ACL 2020) is a BERT-like bidirectional encoder joined to a GPT-like autoregressive decoder. It is pre-trained to reconstruct a document from a corrupted version. The paper compared token masking, token deletion, **text infilling** (spans with Poisson($\lambda=3$) lengths, each replaced by a single [MASK] token, so the model must also predict how many tokens are missing), sentence permutation and document rotation. Text infilling plus sentence permutation worked best. BART-large has 12 encoder and 12 decoder layers of width 1024, uses GELU and learned position embeddings, and was trained like RoBERTa on 160 GB of text. It matched RoBERTa on GLUE and SQuAD, gained up to 6 ROUGE on abstractive summarisation, and added 1.1 BLEU over a back-translation system on WMT Romanian–English.

A useful accounting from the T5 paper: an encoder–decoder with $L$ layers in each stack has about as many parameters as a $2L$-layer decoder-only model. Its compute is roughly that of an $L$-layer one, because each token passes through only one of the two stacks. The cost is an architectural split: the input must be known in advance and is encoded once, which suits conditional tasks and not open-ended continuation (see **Decoder-only transformer**).

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

PAD, BOS, EOS = 0, 1, 2

def sinusoid(L, d):
    pos = torch.arange(L)[:, None]
    ang = pos / 10000 ** (torch.arange(0, d, 2) / d)
    pe = torch.zeros(L, d)
    pe[:, 0::2], pe[:, 1::2] = torch.sin(ang), torch.cos(ang)
    return pe

def causal(T, device):                                   # True = may NOT attend
    return torch.ones(T, T, dtype=torch.bool, device=device).triu(1)

class Seq2SeqTransformer(nn.Module):
    def __init__(self, vocab, d=512, h=8, N=6, d_ff=2048, p=0.1, max_len=1024):
        super().__init__()
        self.d, self.emb = d, nn.Embedding(vocab, d, padding_idx=PAD)   # shared src/tgt/output
        self.register_buffer("pe", sinusoid(max_len, d), persistent=False)
        self.tf = nn.Transformer(d_model=d, nhead=h, num_encoder_layers=N, num_decoder_layers=N,
                                 dim_feedforward=d_ff, dropout=p, batch_first=True)
        self.drop = nn.Dropout(p)

    def embed(self, ids):
        return self.drop(self.emb(ids) * math.sqrt(self.d) + self.pe[: ids.size(1)])

    def forward(self, src, tgt_in):                       # src (N, S), tgt_in (N, T)
        sp, tp = src == PAD, tgt_in == PAD
        z = self.tf(self.embed(src), self.embed(tgt_in), tgt_mask=causal(tgt_in.size(1), src.device),
                    src_key_padding_mask=sp, tgt_key_padding_mask=tp, memory_key_padding_mask=sp)
        return z @ self.emb.weight.T                      # tied output projection -> (N, T, V)

    @torch.no_grad()
    def greedy(self, src, max_new=50):
        sp = src == PAD
        mem = self.tf.encoder(self.embed(src), src_key_padding_mask=sp)       # encoder runs once
        ys = torch.full((src.size(0), 1), BOS, device=src.device)
        for _ in range(max_new):                          # no KV cache here: re-runs the prefix
            z = self.tf.decoder(self.embed(ys), mem, tgt_mask=causal(ys.size(1), src.device),
                                memory_key_padding_mask=sp)
            nxt = (z[:, -1] @ self.emb.weight.T).argmax(-1, keepdim=True)
            ys = torch.cat([ys, nxt], 1)
            if (nxt == EOS).all():
                break
        return ys

V = 37000
model = Seq2SeqTransformer(V)
print(sum(p.numel() for p in model.parameters()))     # 63084544 (nn.Transformer adds 2 final LNs: +2048)

opt = torch.optim.Adam(model.parameters(), lr=1.0, betas=(0.9, 0.98), eps=1e-9)
sched = torch.optim.lr_scheduler.LambdaLR(
    opt, lambda s: 512 ** -0.5 * min((s + 1) ** -0.5, (s + 1) * 4000 ** -1.5))

src = torch.randint(3, V, (2, 10)); src[1, 8:] = PAD
tgt = torch.randint(3, V, (2, 9)); tgt[:, 0] = BOS; tgt[1, 7:] = PAD
tgt_in, tgt_out = tgt[:, :-1], tgt[:, 1:]                 # shift right
logits = model(src, tgt_in)
print(logits.shape)                                       # torch.Size([2, 8, 37000])
loss = F.cross_entropy(logits.flatten(0, 1), tgt_out.flatten(), ignore_index=PAD, label_smoothing=0.1)
loss.backward(); opt.step(); sched.step()

model.eval()
print(model.greedy(src, max_new=5).shape)                 # torch.Size([2, 6]) unless both rows emit EOS early
```

Transformer base for translation: shared embeddings, sinusoidal PE, masks, the paper's schedule, greedy decoding

> **Three masks, three different jobs:** Encoder self-attention needs the source padding mask. Decoder self-attention needs the causal mask, plus the target padding mask if the loss does not already ignore those positions. Cross-attention needs the **source** padding mask again, passed as `memory_key_padding_mask` in PyTorch. Dropping that last one is a common silent bug: decoder queries then read the encoder states of [PAD] tokens, and the output depends on how the batch was padded.

> **PyTorch's nn.Transformer is not exactly the 2017 model:** `nn.Transformer` adds a final LayerNorm after both stacks even in post-LN mode (hence 63,084,544 parameters, 2,048 more than the paper's architecture). It has no embeddings, positional encodings or output layer. Its boolean masks use True for "not allowed", while `F.scaled_dot_product_attention` uses True for "allowed". Check the convention every time you switch APIs.

> **Cross-attention is the only bridge:** The decoder sees the source only through $m$, and every decoder layer reads the same final-layer memory. The encoder can therefore run once per input, cache its $2LSd$ cross-attention keys and values, and serve any number of decoding steps or beams. Compared with the RNN encoder–decoder, there is no fixed-size bottleneck: the decoder can attend to all $S$ source vectors at every layer and step.

> **When to pick encoder–decoder over decoder-only:** If the input is given in full and the output is a different sequence (translation, summarisation, speech-to-text, text-to-SQL), the encoder reads the input bidirectionally and spends its compute once. A decoder-only model treats the input as a prefix under a causal mask, unless it uses a prefix-LM mask. Decoder-only models dominate open-ended generation and very large scale mainly because one stack and one objective (next-token prediction) are simpler to scale and to use for any task.

**Strengths**

- Bidirectional encoding of the input plus exact autoregressive generation of the output
- Fully parallel training: no recurrence in either stack
- Cross-attention to all source states in every decoder layer; no fixed-length bottleneck
- Source encoded once; about $2\times$ the parameters of a single stack at roughly single-stack compute per token

**Limitations**

- Quadratic attention in $S$ and $T$
- Generation is sequential, one decoder pass per output token
- Two stacks and three mask types: more plumbing than decoder-only
- Needs a clear input/output split; less natural for open-ended continuation or chat

**Question:** Which pair of hyperparameters distinguishes Transformer big from Transformer base, apart from dropout and training length?

- A. N = 12 instead of 6, and d_k = 128
- B. d_model = 1024 instead of 512, d_ff = 4096 instead of 2048, and 16 heads instead of 8
- C. A separate vocabulary for source and target
- D. Learned instead of sinusoidal positional encodings

**Answer:** B. d_model = 1024 instead of 512, d_ff = 4096 instead of 2048, and 16 heads instead of 8 — Both models have $N=6$ layers per stack and $d_k=d_v=64$. Big doubles $d_{\text{model}}$ and $d_{ff}$, and doubles the heads so that $d_k=1024/16$ stays 64. It uses $P_{drop}=0.3$ (0.1 for En–Fr) and 300K steps. Parameters grow from 65M to 213M, about $3.3\times$: less than the $4\times$ of the $d^2$ terms, because the embedding grows only as $d$.

**Question:** In an encoder–decoder Transformer, where do the keys and values of decoder layer 3's cross-attention come from?

- A. From encoder layer 3
- B. From the final encoder layer's output, the same for every decoder layer
- C. From decoder layer 2
- D. From the source embeddings

**Answer:** B. From the final encoder layer's output, the same for every decoder layer — All decoder layers attend to the same memory $m$, the output of the last encoder layer, through their own projections $W_K^{(\ell)},W_V^{(\ell)}$. The queries come from the decoder layer's own masked self-attention output. Layer-to-layer connections between the two stacks are not part of the original design.

**Key points**

- Encoder–decoder: $m=\mathrm{Enc}(x)$ with full self-attention; decoder $p(y_i\mid y_{<i},m)$ with causal self-attention and cross-attention to the last-layer $m$ in every layer.
- Transformer (Vaswani et al., NeurIPS 2017): base N = 6, d = 512, d_ff = 2048, h = 8 (65M), 27.3 BLEU En–De; big d = 1024, d_ff = 4096, h = 16 (213M), 28.4 En–De, 41.8 En–Fr.
- Three attention patterns: $S\times S$ full, $T\times T$ causal, $T\times S$ cross; one shared embedding matrix, $\times\sqrt d$, sinusoidal positions.
- Training: Adam (0.9, 0.98, $10^{-9}$), warm-up 4000 then $\propto s^{-0.5}$, dropout 0.1, label smoothing 0.1; decoding with beam 4, α = 0.6.
- T5: text-to-text, span corruption on C4, pre-LN, relative position biases; BART: denoising (text infilling + sentence permutation) with a bidirectional encoder and an autoregressive decoder.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
