Encoder–decoder transformer

The encoder–decoder Transformer (Vaswani et al., 2017) reads the source with a bidirectional self-attention stack and generates the target with a causal decoder that cross-attends to the encoder output in every layer. It set WMT'14 records (28.4 BLEU En–De) and underlies T5 and BART.

16 min read1339 wordscore1 viewsupdated 2026-10-12

Definition

Encoder–decoder transformer is a sequence-to-sequence model built only from attention and feed-forward layers. An encoder of NN layers turns the source x1,…,xSx_1,\dots,x_S into a memory m∈RS×dm\in\mathbb R^{S\times d} using unmasked self-attention. A decoder of NN layers generates y1,…,yTy_1,\dots,y_T one token at a time using causal self-attention over its own prefix, cross-attention to mm in every layer, and a softmax over the vocabulary. It models p(y∣x)=∏ip(yi∣y<i,m)p(y\mid x)=\prod_{i}p(y_i\mid y_{<i},m) and is trained end to end with cross-entropy.

The model is the Transformer of Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, "Attention Is All You Need" (NeurIPS 2017). In 2017 the best translation systems were recurrent or convolutional encoder–decoders with attention (see Sequence-to-sequence (Seq2Seq) and Bahdanau attention). Recurrence forces sequential computation over the positions of a sentence, so the work cannot be parallelised within a training example. The paper kept the encoder–decoder structure and the attention between the two halves, and replaced every recurrent and convolutional layer with self-attention. A single big model trained for 3.5 days on eight P100 GPUs set new records on WMT'14 English→German and English→French at a fraction of the training cost of the previous best models.

The same architecture was later pre-trained as a general text-to-text model, notably in T5 (Raffel et al., JMLR 2020) and BART (Lewis et al., ACL 2020). It remains the standard choice when the input and output are distinct sequences: translation, summarisation, speech recognition (Whisper), and many structured prediction tasks.

The architecture

e0=d E[x]+P1:S,m=EncN∘⋯∘Enc1(e0)z0=d E[⟨s⟩,y1,…,yT−1]+P1:T,zN=DecN(…Dec1(z0,m)…,m)p(yi∣y<i,x)=softmax(zN,i E⊤)\begin{aligned}e_0&=\sqrt d\,E[x]+P_{1:S},\qquad m=\mathrm{Enc}_N\circ\dots\circ\mathrm{Enc}_1(e_0)\\ z_0&=\sqrt d\,E[\langle s\rangle,y_1,\dots,y_{T-1}]+P_{1:T},\qquad z_N=\mathrm{Dec}_N(\dots\mathrm{Dec}_1(z_0,m)\dots,m)\\ p(y_i\mid y_{<i},x)&=\mathrm{softmax}\big(z_{N,i}\,E^\top\big)\end{aligned}
E∈RV×dE\in\mathbb R^{V\times d}
one embedding matrix shared by the source embedding, the target embedding and the output projection (the paper used a shared source–target vocabulary for this)
PP
sinusoidal positional encodings, Pp,2k=sin⁡(p/100002k/d)P_{p,2k}=\sin(p/10000^{2k/d}), Pp,2k+1=cos⁡(p/100002k/d)P_{p,2k+1}=\cos(p/10000^{2k/d}) (see Positional encoding)
Encℓ\mathrm{Enc}_\ell
self-attention + FFN with residuals and post-LN (see Transformer encoder)
Decℓ(z,m)\mathrm{Dec}_\ell(z,m)
masked self-attention, cross-attention to mm, FFN (see Transformer decoder)
mm
the encoder output of the last layer, read by the cross-attention of every decoder layer

Attention appears in three forms, with three mask patterns. In encoder self-attention an S×SS\times S matrix is fully visible except for source padding. In decoder self-attention a T×TT\times T matrix is lower-triangular. In cross-attention a T×ST\times S matrix has queries from the decoder and keys and values from mm, and it is fully visible except for source padding. Each uses hh heads of width d/hd/h (see Multi-head attention) and the scaled dot product softmax(QK⊤/dk)V\mathrm{softmax}(QK^\top/\sqrt{d_k})V (see Scaled dot-product attention).

The paper compared costs per layer. A self-attention layer costs O(n2d)O(n^2d) with O(1)O(1) sequential operations and maximum path length O(1)O(1). A recurrent layer costs O(nd2)O(nd^2) with O(n)O(n) sequential operations and path length O(n)O(n). For sentences, where n<dn<d, self-attention is also cheaper in FLOPs. Every source and target position is connected to every other through a constant number of layers.

modelNd_modeld_ffhd_k = d_vP_dropε_lstrain stepsparams
base651220488640.10.1100K65M
big61024409616640.30.1300K213M
Base and big configurations (Vaswani et al. 2017, Table 3)

Training setup in the paper

Data
WMT 2014 En–De: about 4.5M sentence pairs, byte-pair encoding with a shared source–target vocabulary of about 37,000 tokens. En–Fr: 36M sentences, 32,000 word-piece vocabulary
Batches
sentence pairs batched by approximate length; about 25,000 source and 25,000 target tokens per batch
Hardware
one machine with 8 NVIDIA P100 GPUs; base 0.4 s per step (100K steps, 12 hours), big 1.0 s per step (300K steps, 3.5 days)
Optimiser
Adam, β1=0.9\beta_1=0.9, β2=0.98\beta_2=0.98, ϵ=10−9\epsilon=10^{-9}; lrate=dmodel−0.5min⁡(step−0.5, step⋅warmup−1.5)\mathrm{lrate}=d_{\text{model}}^{-0.5}\min(\text{step}^{-0.5},\ \text{step}\cdot\text{warmup}^{-1.5}) with warmup = 4000
Regularisation
dropout on every sublayer output and on the embedding + PE sums; label smoothing εls=0.1\varepsilon_{ls}=0.1; the big En–Fr model used Pdrop=0.1P_{drop}=0.1
Inference
averaged the last 5 checkpoints (base) or 20 (big); beam size 4, length penalty α = 0.6, max length input + 50

Results

modelEN-DEEN-FRtraining FLOPs (EN-DE)
GNMT + RL24.639.922.3 · 10¹⁹
ConvS2S25.1640.469.6 · 10¹⁸
GNMT + RL ensemble26.3041.16
ConvS2S ensemble26.3641.29
Transformer (base)27.338.13.3 · 10¹⁸
Transformer (big)28.441.82.3 · 10¹⁹
WMT'14 newstest2014 BLEU and estimated training cost (Vaswani et al. 2017, Table 2, selected rows)

The big model beat the best previous results, ensembles included, by more than 2 BLEU on English→German. On English→German the base model already beat all of them, with 3.3⋅10183.3\cdot10^{18} training FLOPs against 9.6⋅10189.6\cdot10^{18} for ConvS2S and 2.3⋅10192.3\cdot10^{19} for GNMT + RL. The paper computed the FLOP estimates as training time × number of GPUs × an assumed sustained 9.5 TFLOPS per P100. The English→French score of the big model is 41.8 in the revised arXiv version. The NeurIPS proceedings version reports 41.0. The same architecture, with 4 layers and d=1024d=1024, also reached 91.3 F1 on WSJ constituency parsing with little task-specific tuning.

Transformer base: N = 6 + 6 layers, d = 512, d_ff = 2048, h = 8, V = 37,000encoder ×6decoder ×6memory (S × d) → K, Vto every decoder layerembedding + PElinear + softmaxp(yᵢ | y<ᵢ, x)feed-forward+ residual, LayerNormcross-attention+ residual, LayerNormmasked self-attn+ residual, LayerNormfeed-forward+ residual, LayerNormself-attention+ residual, LayerNormembedding + PEsource x₁ … x_S<s> y₁ … y_(T−1)(targets shifted right)tiedenc self10×10 → 100dec self8×8 → 36cross8×10 → 80allowed q·k pairs per head per layer: 216rows = queries · cols = keys · click to selectparameters 63,082,496 (paper 65M)embedding (shared)18.9M 30.0%enc self-attn6.3M 10.0%enc FFN12.6M 20.0%dec self-attn6.3M 10.0%cross-attn6.3M 10.0%dec FFN12.6M 20.0%pos. + final LN0 0.0%encoder layer 3,152,384 (12d² = 3,145,728)decoder layer 4,204,032 (16d² = 4,194,304)post-LN, sinusoidal PE, ReLU, ~37K shared BPEcross-attention (decoder): 1,051,648 per layer, 6.3M total, 10.0%queries from the decoder, keys and values from the encoder output
model
fig.The full model, with the three attention patterns for a source of S = 10 and a target of T = 8 tokens: 100 allowed query–key pairs in encoder self-attention, 36 in causal decoder self-attention and 80 in cross-attention, 216 per head and layer. Click a box or a pattern to see its share of the parameters. These are exact counts with biases and LayerNorms. Transformer base comes to 63,082,496 (paper: 65M; the exact vocabulary size is not given). Of that, 30.0% is the shared 37K × 512 embedding, 20.0% each FFN stack and 10.0% each attention type: an encoder layer has 3,152,384 parameters, a decoder layer 4,204,032. Transformer big gives 214,245,376 (paper: 213M). T5-base, with no biases, scale-only LayerNorms, relative position biases and a 32,128 vocabulary, gives 222,903,552 (paper: 220M). BART-large, with learned positions and a 50,265 vocabulary, gives 406,291,456. Its embedding share falls to 12.7%, and each FFN stack is 24.8%.

Worked example: counting Transformer base and its schedule

componentformulaparametersshare
shared embedding / outputV·d = 37,000 · 51218,944,00030.0%
encoder, 6 layers6 · (12d² + 13d)18,914,30430.0%
decoder self-attention + LN6 · (4d² + 4d + 2d)6,309,88810.0%
decoder cross-attention + LN6 · (4d² + 4d + 2d)6,309,88810.0%
decoder FFN + LN6 · (8d² + 5d + 2d)12,604,41620.0%
total63,082,496100%
Exact parameter count of Transformer base with a 37,000-token shared vocabulary (computed)

Parameters, compute and learning rate

  1. 1Per layer with d=512d=512, dff=2048d_{ff}=2048: encoder 12d2+13d=3,152,38412d^2+13d=3{,}152{,}384, decoder 16d2+19d=4,204,03216d^2+19d=4{,}204{,}032. Six of each give 44,138,49644{,}138{,}496, and the embedding adds 18,944,00018{,}944{,}000, for 63.0863.08M. The paper reports 65M. The difference most likely comes from the exact BPE vocabulary, which is given only as "about 37000".
  2. 2Training cost of base: 12 h×3600×8 GPUs×9.5⋅1012=3.28⋅101812\ \text{h}\times3600\times8\ \text{GPUs}\times9.5\cdot10^{12}=3.28\cdot10^{18} FLOPs, the 3.3⋅10183.3\cdot10^{18} of Table 2. Big: 3.5×86,400×8×9.5⋅1012=2.30⋅10193.5\times86{,}400\times8\times9.5\cdot10^{12}=2.30\cdot10^{19}.
  3. 3Learning rate: lrate(s)=512−0.5min⁡(s−0.5, s⋅4000−1.5)\mathrm{lrate}(s)=512^{-0.5}\min(s^{-0.5},\,s\cdot4000^{-1.5}) rises linearly to its peak at s=4000s=4000, 0.044194×0.015811=6.99⋅10−40.044194\times0.015811=6.99\cdot10^{-4}. Then it decays as s−0.5s^{-0.5}: 3.49⋅10−43.49\cdot10^{-4} at 16,00016{,}000 and 1.40⋅10−41.40\cdot10^{-4} at 100,000100{,}000.
  4. 4Allowed query–key pairs per head and layer for S=10S=10, T=8T=8: S2=100S^2=100 (encoder) + T(T+1)/2=36+\ T(T+1)/2=36 (decoder) + TS=80+\ TS=80 (cross) =216=216.

T5 and BART: pre-trained encoder–decoders

T5 (Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li and Liu, "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", JMLR 2020) casts every task as text in, text out: "translate English to German: …", "summarize: …", or a GLUE label written as a word. It pre-trains on C4, a cleaned Common Crawl corpus of about 750 GB, with span corruption. 15% of tokens are dropped in spans of mean length 3. Each span is replaced by a sentinel token in the input, and the target lists the dropped spans after their sentinels. In the paper's controlled comparison, the encoder–decoder with this denoising objective beat decoder-only and prefix-LM variants in comparable settings. Architecturally T5 changes the 2017 model in small ways: pre-LN with a scale-only LayerNorm, no bias terms, and relative position biases (32 buckets, a learned scalar per head added to the attention logits) in place of sinusoids. The released sizes were Small (60M), Base (220M), Large (770M), 3B and 11B.

BART (Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov and Zettlemoyer, "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension", ACL 2020) is a BERT-like bidirectional encoder joined to a GPT-like autoregressive decoder. It is pre-trained to reconstruct a document from a corrupted version. The paper compared token masking, token deletion, text infilling (spans with Poisson(λ=3\lambda=3) lengths, each replaced by a single [MASK] token, so the model must also predict how many tokens are missing), sentence permutation and document rotation. Text infilling plus sentence permutation worked best. BART-large has 12 encoder and 12 decoder layers of width 1024, uses GELU and learned position embeddings, and was trained like RoBERTa on 160 GB of text. It matched RoBERTa on GLUE and SQuAD, gained up to 6 ROUGE on abstractive summarisation, and added 1.1 BLEU over a back-translation system on WMT Romanian–English.

A useful accounting from the T5 paper: an encoder–decoder with LL layers in each stack has about as many parameters as a 2L2L-layer decoder-only model. Its compute is roughly that of an LL-layer one, because each token passes through only one of the two stacks. The cost is an architectural split: the input must be known in advance and is encoded once, which suits conditional tasks and not open-ended continuation (see Decoder-only transformer).

python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3PAD, BOS, EOS = 0, 1, 2
4
5def sinusoid(L, d):
6 pos = torch.arange(L)[:, None]
7 ang = pos / 10000 ** (torch.arange(0, d, 2) / d)
8 pe = torch.zeros(L, d)
9 pe[:, 0::2], pe[:, 1::2] = torch.sin(ang), torch.cos(ang)
10 return pe
11
12def causal(T, device): # True = may NOT attend
13 return torch.ones(T, T, dtype=torch.bool, device=device).triu(1)
14
15class Seq2SeqTransformer(nn.Module):
16 def __init__(self, vocab, d=512, h=8, N=6, d_ff=2048, p=0.1, max_len=1024):
17 super().__init__()
18 self.d, self.emb = d, nn.Embedding(vocab, d, padding_idx=PAD) # shared src/tgt/output
19 self.register_buffer("pe", sinusoid(max_len, d), persistent=False)
20 self.tf = nn.Transformer(d_model=d, nhead=h, num_encoder_layers=N, num_decoder_layers=N,
21 dim_feedforward=d_ff, dropout=p, batch_first=True)
22 self.drop = nn.Dropout(p)
23
24 def embed(self, ids):
25 return self.drop(self.emb(ids) * math.sqrt(self.d) + self.pe[: ids.size(1)])
26
27 def forward(self, src, tgt_in): # src (N, S), tgt_in (N, T)
28 sp, tp = src == PAD, tgt_in == PAD
29 z = self.tf(self.embed(src), self.embed(tgt_in), tgt_mask=causal(tgt_in.size(1), src.device),
30 src_key_padding_mask=sp, tgt_key_padding_mask=tp, memory_key_padding_mask=sp)
31 return z @ self.emb.weight.T # tied output projection -> (N, T, V)
32
33 @torch.no_grad()
34 def greedy(self, src, max_new=50):
35 sp = src == PAD
36 mem = self.tf.encoder(self.embed(src), src_key_padding_mask=sp) # encoder runs once
37 ys = torch.full((src.size(0), 1), BOS, device=src.device)
38 for _ in range(max_new): # no KV cache here: re-runs the prefix
39 z = self.tf.decoder(self.embed(ys), mem, tgt_mask=causal(ys.size(1), src.device),
40 memory_key_padding_mask=sp)
41 nxt = (z[:, -1] @ self.emb.weight.T).argmax(-1, keepdim=True)
42 ys = torch.cat([ys, nxt], 1)
43 if (nxt == EOS).all():
44 break
45 return ys
46
47V = 37000
48model = Seq2SeqTransformer(V)
49print(sum(p.numel() for p in model.parameters())) # 63084544 (nn.Transformer adds 2 final LNs: +2048)
50
51opt = torch.optim.Adam(model.parameters(), lr=1.0, betas=(0.9, 0.98), eps=1e-9)
52sched = torch.optim.lr_scheduler.LambdaLR(
53 opt, lambda s: 512 ** -0.5 * min((s + 1) ** -0.5, (s + 1) * 4000 ** -1.5))
54
55src = torch.randint(3, V, (2, 10)); src[1, 8:] = PAD
56tgt = torch.randint(3, V, (2, 9)); tgt[:, 0] = BOS; tgt[1, 7:] = PAD
57tgt_in, tgt_out = tgt[:, :-1], tgt[:, 1:] # shift right
58logits = model(src, tgt_in)
59print(logits.shape) # torch.Size([2, 8, 37000])
60loss = F.cross_entropy(logits.flatten(0, 1), tgt_out.flatten(), ignore_index=PAD, label_smoothing=0.1)
61loss.backward(); opt.step(); sched.step()
62
63model.eval()
64print(model.greedy(src, max_new=5).shape) # torch.Size([2, 6]) unless both rows emit EOS early
Transformer base for translation: shared embeddings, sinusoidal PE, masks, the paper's schedule, greedy decoding

Use when

  • +Bidirectional encoding of the input plus exact autoregressive generation of the output
  • +Fully parallel training: no recurrence in either stack
  • +Cross-attention to all source states in every decoder layer; no fixed-length bottleneck
  • +Source encoded once; about 2×2\times the parameters of a single stack at roughly single-stack compute per token

Avoid when

  • −Quadratic attention in SS and TT
  • −Generation is sequential, one decoder pass per output token
  • −Two stacks and three mask types: more plumbing than decoder-only
  • −Needs a clear input/output split; less natural for open-ended continuation or chat

Check yourself

Which pair of hyperparameters distinguishes Transformer big from Transformer base, apart from dropout and training length?

Check yourself

In an encoder–decoder Transformer, where do the keys and values of decoder layer 3's cross-attention come from?

Remember

  • ✓Encoder–decoder: m=Enc(x)m=\mathrm{Enc}(x) with full self-attention; decoder p(yi∣y<i,m)p(y_i\mid y_{<i},m) with causal self-attention and cross-attention to the last-layer mm in every layer.
  • ✓Transformer (Vaswani et al., NeurIPS 2017): base N = 6, d = 512, d_ff = 2048, h = 8 (65M), 27.3 BLEU En–De; big d = 1024, d_ff = 4096, h = 16 (213M), 28.4 En–De, 41.8 En–Fr.
  • ✓Three attention patterns: S×SS\times S full, T×TT\times T causal, T×ST\times S cross; one shared embedding matrix, ×d\times\sqrt d, sinusoidal positions.
  • ✓Training: Adam (0.9, 0.98, 10−910^{-9}), warm-up 4000 then ∝s−0.5\propto s^{-0.5}, dropout 0.1, label smoothing 0.1; decoding with beam 4, α = 0.6.
  • ✓T5: text-to-text, span corruption on C4, pre-LN, relative position biases; BART: denoising (text infilling + sentence permutation) with a bidirectional encoder and an autoregressive decoder.
← back to Deep Learning roadmap