Encoder–decoder models

Encoder–decoder LMs (T5, BART, mT5, Flan-T5, Whisper) read the input once with a bidirectional encoder and write with a decoder that cross-attends to it. Each token crosses one stack, so they hold ~2× the parameters of a decoder at similar per-token compute. They suit fixed, long inputs.

14 min read1636 wordscore0 viewsupdated 2026-10-13

Definition

Encoder–decoder model is a pre-trained language model with two transformer stacks. A bidirectional encoder reads the whole input xx once. A causal decoder generates the output yy token by token and reads the encoder's output through Cross-attention in every layer. The model is trained on p(y∣x)p(y\mid x), usually first with a denoising objective on unlabeled text and then on tasks. The pre-trained family begins in 2019–2020 with MASS (Song et al., ICML 2019), BART (Lewis et al., ACL 2020) and T5 (Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", JMLR 2020). The architecture itself, its masks and its parameter counts are covered in Encoder–decoder transformer. This article is about the model family and when it is the right choice.

Encoder–decoders came first. Sequence-to-sequence (Seq2Seq) RNNs (Sutskever, Vinyals and Le, 2014) and the original Transformer (Vaswani et al., 2017) were built for translation, where the input and output are different sequences. When pre-training arrived, BERT took the encoder and GPT took the decoder. T5 and BART put the two halves back together and pre-trained the whole model on corrupted text. T5's controlled study found the encoder–decoder with a denoising objective the strongest of its architectures for fine-tuned transfer. For a few years T5 checkpoints were the default for summarisation, question answering and translation research.

Decoder-only models then won at the largest scales, for reasons covered in Decoder-only models. Encoder–decoders did not disappear. They remain standard where the input is complete before generation starts, is much longer than the output, or comes in a different modality. Speech recognition (Whisper), large-scale machine translation (NLLB-200, a 54.5B-parameter mixture-of-experts encoder–decoder covering 200 languages) and compact task-specific models (Flan-T5) are the main cases.

The objective

pθ(y∣x)=∏t=1Opθ(yt∣y<t, m),m=Encθ(x)∈RI×d,L=−∑t=1Olog⁡pθ(yt∣y<t, Encθ(x~))p_\theta(y\mid x)=\prod_{t=1}^{O}p_\theta\big(y_t\mid y_{<t},\,m\big),\qquad m=\mathrm{Enc}_\theta(x)\in\mathbb R^{I\times d},\qquad \mathcal L=-\sum_{t=1}^{O}\log p_\theta(y_t\mid y_{<t},\,\mathrm{Enc}_\theta(\tilde x))
mm
the memory: one dd-dimensional vector per input token, computed once, read by every decoder layer
I, OI,\ O
input and output lengths
x~\tilde x
the corrupted input in pre-training: T5 drops 15% of tokens in spans of mean length 3 (T5); BART masks Poisson-length spans and shuffles sentences (BART). In fine-tuning, x~=x\tilde x=x is the task input
yy
the target: the missing spans (T5), the original document (BART), or the task answer

The loss covers only the OO target tokens. The input is never predicted, so it can be encoded with full bidirectional attention: every input token sees every other. This is the structural difference from a decoder-only model, where the input is a causal prefix and each token sees only those before it.

read once, bidirectionally; write token by tokenencoderruns once per input× 24 layersself-attentionfull: all I² pairsMLP1024 → 4096 → 1024input: I tokenstext, or audio framesmemory m: I × 1024decoderruns once per output token× 24 layerscausal self-attncross-attentionqueries → all of mMLP1024 → 4096 → 1024K, Vy₁ … y_(t−1)output so farsoftmax → y_tshape shown: T5-Large, 24 + 24 layers, d = 1024, 16 heads, ~770Meach token passes through one stack: ~2× the parameters of a24-layer decoder for about the same compute per tokenthe familyT5 · 2020C4, span corruption, text-to-text60M, 220M, 770M, 3B, 11BBART · 2020denoising: text infilling + shufflingbase 140M, large 400MmT5 · 2021mC4, 101 languages, 250K vocab300M to 13BFlan-T5 · 2022T5 + instruction tuning, 1,836 tasks80M to 11BWhisper · 2022680K h of audio → transcripts39M to 1.55B
fig.Left: the shape shared by the family, drawn with T5-Large dimensions (24 + 24 layers, d = 1024, d_ff = 4096, 16 heads, about 770M parameters). The encoder runs once over all I input tokens with unmasked self-attention and produces the memory m. Each decoder layer adds a cross-attention sublayer whose keys and values come from m, so the memory is projected once per input and reused at every output step. Right: the main pre-trained encoder–decoders and what they were trained on.

The family

T5 casts every task as text to text with a task prefix, for example "summarize: …". It was pre-trained on C4, about 750 GB of cleaned Common Crawl, and released in five sizes from 60M to 11B. T5.1.1 changed the recipe: GEGLU feed-forward layers, no parameter sharing between the embedding and the output layer, no dropout during pre-training, and C4 only. The LM-adapted T5.1.1 checkpoints were trained for a further 100K steps with a plain language-modelling objective. They were used for prompt tuning (Lester et al., 2021) and as the starting point of T0.

BART pairs a BERT-style encoder with a GPT-style decoder and learns to reconstruct documents from text infilling and sentence shuffling. mBART (Liu et al., 2020) applied the same objective to 25 languages for translation. mT5 (Xue et al., "mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer", NAACL 2021) trained T5.1.1 on mC4, covering 101 languages, with a 250K SentencePiece vocabulary, in sizes from 300M to 13B. ByT5 (Xue et al., TACL 2022) dropped the tokenizer altogether, read UTF-8 bytes and made the encoder three times deeper than the decoder.

T0 (Sanh et al., ICLR 2022) and Flan-T5 (Chung et al., "Scaling Instruction-Finetuned Language Models", 2022) showed that instruction tuning turns T5 into a zero-shot model. Flan-T5 was fine-tuned on a collection of 1,836 tasks, many phrased with chain-of-thought or few-shot templates. UL2 (Tay et al., ICLR 2023) pre-trained a 20B encoder–decoder on a mixture of denoisers. Whisper (Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision", 2022) is the clearest modern case. Its encoder reads 30 seconds of 80-channel log-Mel spectrogram, downsampled by two convolution layers to 1,500 positions. Its decoder writes the transcript, with special tokens choosing the language and the task (transcribe or translate). It was trained on 680,000 hours of labelled audio.

Configurations from the papers

T5-11B
24 encoder + 24 decoder layers, dmodel=1024d_{\text{model}}=1024, dff=65536d_{ff}=65536, 128 heads with dkv=128d_{kv}=128; the 3B model uses dff=16384d_{ff}=16384 and 32 heads
T5 pre-training
219=524,2882^{19}=524{,}288 steps, batch 128 sequences of 512 tokens (2162^{16} tokens), about 34B tokens; 32K SentencePiece vocabulary; relative position biases with 32 buckets
BART-large
12 + 12 layers, d = 1024, about 400M parameters, pre-trained on 160 GB of text with RoBERTa's batch size and data
mT5
mC4, 101 languages, 250K-token vocabulary, Small 300M to XXL 13B; at small sizes the embeddings are most of the parameters (mT5-Small 300M against T5-Small 60M)
Flan-T5
T5 checkpoints from 80M to 11B, instruction-tuned on 1,836 tasks
Whisper large
32 + 32 layers, width 1280, 20 heads, 1.55B parameters; encoder input 3,000 Mel frames → 1,500 positions; decoder context 448 tokens; sizes 39M (tiny) to 1.55B

Compute per token, compared with decoder-only

Take an encoder–decoder with LL layers per stack and width dd, and set dff=4dd_{ff}=4d. An encoder layer has 12d212d^2 weights. A decoder layer has 16d216d^2, since cross-attention adds four more d×dd\times d projections. Each input token passes only through the encoder and each output token only through the decoder. So the model holds 28Ld228Ld^2 parameters, yet a token meets only 12d212d^2 to 14d214d^2 of them per layer. Counting 2 FLOPs per multiply–add and 4d4d per attended query–key pair:

FED=L[24d2I+4dI2⏟encoder+4d2I⏟cross K,V]+L[28d2O⏟decoder weights+2dO(O+1)⏟causal self+4dIO⏟cross],FD=LD[24d2(I+O)+2d (I+O)(I+O+1)]F_{\text{ED}}=L\big[\underbrace{24d^2I+4dI^2}_{\text{encoder}}+\underbrace{4d^2I}_{\text{cross }K,V}\big]+L\big[\underbrace{28d^2O}_{\text{decoder weights}}+\underbrace{2dO(O+1)}_{\text{causal self}}+\underbrace{4dIO}_{\text{cross}}\big],\qquad F_{\text{D}}=L_D\big[24d^2(I+O)+2d\,(I+O)(I+O+1)\big]
24d224d^2
2×12d22\times12d^2: the encoder layer's weights applied to one token; the decoder applies 2×14d22\times14d^2 (cross K,VK,V projections act on mm instead)
4dI24dI^2
bidirectional encoder attention over all I2I^2 pairs: twice the 2dI(I+1)2dI(I+1) a causal prefix pays
LDL_D
decoder-only depth; LD=2812LL_D=\tfrac{28}{12}L matches parameters, LD=LL_D=L matches the depth each token sees
KV\text{KV}
cache: 2Ld(I+O)2Ld(I+O) numbers for the encoder–decoder (cross + self) against 2LDd(I+O)2L_Dd(I+O)

Two comparisons follow. Take first a decoder-only model with the same parameters (LD=56L_D=56 for L=24L=24). It needs 1.3–2.1 times the FLOPs of the encoder–decoder (about 2× unless the input is very long), a cache 56/24=2.3356/24=2.33 times larger, and twice the weights read per generated token. Decoding is memory-bound, so that last factor is close to a 2× speed-up for the encoder–decoder. Against a decoder with the same depth per token (LD=24L_D=24), the encoder–decoder has 2.33 times the parameters for 1.1–1.8 times the FLOPs. The upper end comes from long inputs, where full encoder attention costs twice what a causal prefix pays.

This is why Wang et al. ("What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?", ICML 2022) compared architectures at matched compute rather than matched size. Their encoder–decoders had about twice the parameters of their decoder-only models. Under that matching, the causal decoder with a plain LM objective was best at zero-shot use straight after pre-training. Encoder–decoder and non-causal decoders trained with masked denoising and then multitask-finetuned were best overall. Pre-training objective and architecture interact: an encoder–decoder pre-trained with span corruption needs multitask or instruction tuning before it can be prompted, which is exactly what T0 and Flan-T5 add.

one example: I = 4,096 input, O = 256 output tokens · d = 1024, d_ff = 4d■ enc–dec 24+24■ dec-only 56L■ dec-only 24Lforward FLOPsenc–dec 24+244.82 TFLOPdec-only 56L8.31 TFLOP (1.72×)dec-only 24L3.56 TFLOP (0.74×)KV cache (bf16)enc–dec 24+24408.0 MiBdec-only 56L952.0 MiB (2.33×)dec-only 24L408.0 MiB (1.00×)weights read / tokenenc–dec 24+24352Mdec-only 56L705M (2.00×)dec-only 24L302M (0.86×)parametersenc–dec 24+24705Mdec-only 56L705M (1.00×)dec-only 24L302M (0.43×)FLOP bars: solid = encode / prefill of the input, pale = generating the outputvs the same-size decoder: 1.72× FLOPs, 2.33× cache, 2.00× weights per token for decoder-onlyvs the same-depth decoder: enc–dec has 2.33× the parameters for 1.35× the FLOPsencoder attention over all I² pairs: 34.2% of enc–dec FLOPs (causal prefill pays half)training (forward + backward ≈ 3× forward): enc–dec 14.47 TFLOP, dec-only 56L 24.92 TFLOPinput ≫ output: input read once, bidirectionally; decoding reads only the decoder halfno embeddings · cross K, V computed once from the encoder output · 2 FLOPs per multiply–add
task
fig.Forward FLOPs, KV cache, weights read per generated token and parameters for one example. The models are an encoder–decoder with 24 + 24 layers (T5-Large shape, 705M non-embedding parameters), a decoder-only model with 56 layers (the same 705M), and one with 24 layers (302M, the same depth per token), all at d = 1024. At the default summarisation setting, 4,096 → 256, the encoder–decoder needs 4.82 TFLOP and a 408 MiB cache. The same-size decoder needs 8.31 TFLOP (1.72×) and 952 MiB (2.33×), and reads 705M rather than 352M weights per token (2.00×). The 24-layer decoder needs 3.56 TFLOP (0.74×), so the encoder–decoder's 2.33× parameters cost 1.35× compute. Encoder attention is 34.2% of its FLOPs. For translation (64 → 64) the ratios are 2.00× and 0.86×, and encoder attention is 0.4%. For long-document QA (32K → 64) they are 1.32× and 0.56×, and encoder attention is 81.9%: full attention over 32K tokens is the price of reading them bidirectionally. For a chat reply (256 → 4K) the ratios are 2.08× and 0.89×. Most of the work is in the decoder, and the split buys little.

When the encoder–decoder wins

The input is given and much longer than the output. Summarisation, document QA, classification phrased as generation, and retrieval-augmented reading all fit. The encoder reads the input once, bidirectionally, and every output step then reads only the decoder half of the weights. Fusion-in-Decoder (Izacard and Grave, EACL 2021) pushes this further. It encodes each retrieved passage independently with its question, concatenates the memories, and lets the decoder cross-attend to all of them. Encoder cost then grows linearly with the number of passages rather than quadratically with their total length.

Input and output are different kinds of sequence. In translation the source language is read and the target language is written, so separate weights for reading and writing are a natural inductive bias. In speech recognition the input is not text at all. Whisper's encoder handles 1,500 audio positions, its decoder a few hundred text tokens. The modality split matches the architectural split. The same holds for image captioning and OCR models with vision encoders.

It loses when the boundary between input and output is fluid. In multi-turn chat, each reply becomes part of the next input. An encoder–decoder must then re-encode the whole history, because new tokens change every bidirectional representation, while a decoder-only model just extends its KV cache. In-context learning, open-ended continuation and agentic loops follow the same pattern. Ecosystem effects compound the gap: nearly all frontier-scale training, tooling and inference work since 2021 has targeted decoder-only models.

Worked example: one summarisation example, counted

Take a small encoder–decoder with L=6L=6 layers per stack and d=512d=512 (the shape of Transformer base), an input of I=1000I=1000 tokens and an output of O=100O=100. Compare it with a 14-layer decoder-only model, which has the same 28⋅6=16828\cdot6=168 units of d2d^2 in parameters, and with a 6-layer one.

termformulaGFLOP
encoder weightsL · 24d² · I37.75
encoder attentionL · 4d · I²12.29
cross K, V of the memoryL · 4d² · I6.29
decoder weightsL · 28d² · O4.40
decoder causal self-attentionL · 2d · O(O + 1)0.06
cross-attentionL · 4d · I · O1.23
encoder–decoder total62.02
decoder-only, 14 layers (same parameters)14 · [24d² · 1100 + 2d · 1100 · 1101]114.25
decoder-only, 6 layers (same depth)6 · [24d² · 1100 + 2d · 1100 · 1101]48.96
Forward FLOPs term by term (computed in Python; 1 GFLOP = 10⁹)

Reading the numbers

  1. 1Parameters: encoder–decoder 28⋅6⋅5122=44,040,19228\cdot6\cdot512^2=44{,}040{,}192, the same as the 14-layer decoder (14⋅12⋅512214\cdot12\cdot512^2). The 6-layer decoder has 18,874,36818{,}874{,}368.
  2. 2The input dominates: the encoder side (37.75+12.29+6.29=56.3337.75+12.29+6.29=56.33 GFLOP) is 91% of the total, because I=10 OI=10\,O.
  3. 3Against the same-size decoder: 114.25/62.02=1.84114.25/62.02=1.84 times the FLOPs for the decoder-only model. Against the same-depth decoder: the encoder–decoder costs 62.02/48.96=1.2762.02/48.96=1.27 times as much, for 44.0/18.9=2.3344.0/18.9=2.33 times the parameters.
  4. 4KV cache: encoder–decoder 2⋅6⋅512⋅(1000+100)=6,758,4002\cdot6\cdot512\cdot(1000+100)=6{,}758{,}400 numbers (13.5 MB in bf16), the same as the 6-layer decoder. The 14-layer decoder needs 15,769,60015{,}769{,}600 (2.33×).
  5. 5Weights read per generated token: 14⋅6⋅5122=22.014\cdot6\cdot512^2=22.0M for the encoder–decoder against 44.044.0M for the same-size decoder. Memory-bound decoding is about twice as fast.
python
1def flops_encdec(I, O, L, d):
2 """Forward FLOPs for one example: encoder over I tokens, decoder over O tokens."""
3 encode = L * (24 * d * d * I + 4 * d * I * I) # 12d² weights, all I² attention pairs
4 cross_kv = L * 4 * d * d * I # K, V of the memory, once per input
5 decode = L * (28 * d * d * O + 2 * d * O * (O + 1) + 4 * d * I * O)
6 return encode + cross_kv + decode
7
8def flops_decoder(I, O, L, d):
9 T = I + O # prompt and answer in one causal stream
10 return L * (24 * d * d * T + 2 * d * T * (T + 1))
11
12d, L = 1024, 24 # T5-Large shape: 24 + 24 layers
13for I, O in [(64, 64), (4096, 256), (32768, 64), (256, 4096)]:
14 ed = flops_encdec(I, O, L, d)
15 same_params = flops_decoder(I, O, 56, d) # 56 · 12d² = 24 · (12 + 16) d²
16 same_depth = flops_decoder(I, O, 24, d)
17 print(f"{I:>6} -> {O:<5} enc-dec {ed / 1e12:6.2f} TFLOP | "
18 f"dec-56L {same_params / ed:.2f}x | dec-24L {same_depth / ed:.2f}x")
19# 64 -> 64 enc-dec 0.09 TFLOP | dec-56L 2.00x | dec-24L 0.86x
20# 4096 -> 256 enc-dec 4.82 TFLOP | dec-56L 1.72x | dec-24L 0.74x
21# 32768 -> 64 enc-dec 128.89 TFLOP | dec-56L 1.32x | dec-24L 0.56x
22# 256 -> 4096 enc-dec 4.00 TFLOP | dec-56L 2.08x | dec-24L 0.89x
23
24# --- Using real encoder–decoders with Hugging Face transformers ---
25import torch
26from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline
27
28tok = AutoTokenizer.from_pretrained("google/flan-t5-base")
29model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base").eval()
30
31doc = "summarize: " + open("report.txt").read()
32enc_in = tok(doc, return_tensors="pt", truncation=True, max_length=2048)
33with torch.no_grad():
34 memory = model.get_encoder()(**enc_in) # the encoder runs once: m has shape (1, I, 768)
35 greedy = model.generate(encoder_outputs=memory, attention_mask=enc_in.attention_mask, max_new_tokens=80)
36 sampled = model.generate(encoder_outputs=memory, attention_mask=enc_in.attention_mask,
37 max_new_tokens=80, do_sample=True, top_p=0.9)
38print(tok.batch_decode(greedy, skip_special_tokens=True)[0])
39
40# Training: labels are the target; the decoder input is the labels shifted right internally.
41labels = tok("A two-sentence summary.", return_tensors="pt").input_ids
42loss = model(**enc_in, labels=labels).loss # cross-entropy over the O target tokens only
43
44# Whisper: 30 s of audio -> log-Mel -> 1,500 encoder positions -> text tokens.
45asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
46print(asr("meeting.wav", return_timestamps=True)["text"])
The FLOP model of the figure (plain Python, run; output in comments), then Flan-T5 with the encoder run once and two decodes reusing it, and Whisper through the pipeline API (transformers; not run here)

Use when

  • +Bidirectional reading of the input: every input token sees the whole input in every layer
  • +About 2× the parameters of a decoder at similar per-token compute, and about half the weights read per generated token at equal size
  • +Smaller KV cache at equal size; cross-attention keys and values are computed once per input
  • +A natural fit for conditional tasks and modality changes: translation, summarisation, speech recognition, captioning
  • +Fusion-in-Decoder style reading of many passages at linear encoder cost

Avoid when

  • −Needs a fixed split into input and output; growing inputs (chat, agents) force a full re-encode
  • −Denoising-pretrained checkpoints need multitask or instruction tuning before zero-shot prompting works
  • −Full encoder attention costs twice a causal prefix at long inputs
  • −Two stacks, three attention types and a smaller ecosystem than decoder-only models at frontier scale

Check yourself

An encoder–decoder with 24 + 24 layers and a decoder-only model with 56 layers have the same parameter count. When generating a 256-token summary of a 4,096-token document, why is the encoder–decoder faster per generated token?

Check yourself

Fusion-in-Decoder answers a question from 100 retrieved passages of 200 tokens each. Why does it encode the passages independently rather than as one 20,000-token input?

Remember

  • ✓Encoder–decoder LMs model p(y∣x)=∏tp(yt∣y<t,Enc(x))p(y\mid x)=\prod_t p(y_t\mid y_{<t},\mathrm{Enc}(x)): a bidirectional encoder run once, a causal decoder with cross-attention, and a loss only on the output.
  • ✓Family: T5 (C4, span corruption, 60M–11B), BART (denoising), mT5 (101 languages), Flan-T5 (1,836 instruction tasks), UL2, Whisper (680K hours of speech), NLLB-200 (translation).
  • ✓Each token crosses one stack: 28Ld228Ld^2 parameters, but only 1212–14 d214\,d^2 per layer per token. That is 2.33× the parameters of a same-depth decoder for 1.1–1.8× its FLOPs, or half the weights per decode step at equal size.
  • ✓It wins when the input is fixed and long relative to the output, or in a different modality. It loses when the input keeps growing, as in chat and agents, which forces a re-encode.
  • ✓Wang et al. (2022): causal decoders are best zero-shot straight after pre-training; non-causal models with denoising are best after multitask finetuning.
← back to Deep Learning roadmap