# Encoder–decoder models

> Encoder–decoder LMs (T5, BART, mT5, Flan-T5, Whisper) read the input once with a bidirectional encoder and write with a decoder that cross-attends to it. Each token crosses one stack, so they hold ~2× the parameters of a decoder at similar per-token compute. They suit fixed, long inputs.

Source: https://www.mlfoundry.in/learn/encoder-decoder-models  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**Encoder–decoder model:** is a pre-trained language model with two transformer stacks. A bidirectional **encoder** reads the whole input $x$ once. A causal **decoder** generates the output $y$ token by token and reads the encoder's output through **Cross-attention** in every layer. The model is trained on $p(y\mid x)$, usually first with a denoising objective on unlabeled text and then on tasks. The pre-trained family begins in 2019–2020 with MASS (Song et al., ICML 2019), BART (Lewis et al., ACL 2020) and T5 (Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", JMLR 2020). The architecture itself, its masks and its parameter counts are covered in **Encoder–decoder transformer**. This article is about the model family and when it is the right choice.

Encoder–decoders came first. **Sequence-to-sequence (Seq2Seq)** RNNs (Sutskever, Vinyals and Le, 2014) and the original Transformer (Vaswani et al., 2017) were built for translation, where the input and output are different sequences. When pre-training arrived, BERT took the encoder and GPT took the decoder. T5 and BART put the two halves back together and pre-trained the whole model on corrupted text. T5's controlled study found the encoder–decoder with a denoising objective the strongest of its architectures for fine-tuned transfer. For a few years T5 checkpoints were the default for summarisation, question answering and translation research.

Decoder-only models then won at the largest scales, for reasons covered in **Decoder-only models**. Encoder–decoders did not disappear. They remain standard where the input is complete before generation starts, is much longer than the output, or comes in a different modality. Speech recognition (Whisper), large-scale machine translation (NLLB-200, a 54.5B-parameter mixture-of-experts encoder–decoder covering 200 languages) and compact task-specific models (Flan-T5) are the main cases.

## The objective

$$
p_\theta(y\mid x)=\prod_{t=1}^{O}p_\theta\big(y_t\mid y_{<t},\,m\big),\qquad m=\mathrm{Enc}_\theta(x)\in\mathbb R^{I\times d},\qquad \mathcal L=-\sum_{t=1}^{O}\log p_\theta(y_t\mid y_{<t},\,\mathrm{Enc}_\theta(\tilde x))
$$

- $m$: the memory: one $d$-dimensional vector per input token, computed once, read by every decoder layer
- $I,\ O$: input and output lengths
- $\tilde x$: the corrupted input in pre-training: T5 drops 15% of tokens in spans of mean length 3 (**T5**); BART masks Poisson-length spans and shuffles sentences (**BART**). In fine-tuning, $\tilde x=x$ is the task input
- $y$: the target: the missing spans (T5), the original document (BART), or the task answer

The loss covers only the $O$ target tokens. The input is never predicted, so it can be encoded with full bidirectional attention: every input token sees every other. This is the structural difference from a decoder-only model, where the input is a causal prefix and each token sees only those before it.

> Figure: Left: the shape shared by the family, drawn with T5-Large dimensions (24 + 24 layers, d = 1024, d_ff = 4096, 16 heads, about 770M parameters). The encoder runs once over all I input tokens with unmasked self-attention and produces the memory m. Each decoder layer adds a cross-attention sublayer whose keys and values come from m, so the memory is projected once per input and reused at every output step. Right: the main pre-trained encoder–decoders and what they were trained on.

## The family

**T5** casts every task as text to text with a task prefix, for example "summarize: …". It was pre-trained on C4, about 750 GB of cleaned Common Crawl, and released in five sizes from 60M to 11B. **T5.1.1** changed the recipe: GEGLU feed-forward layers, no parameter sharing between the embedding and the output layer, no dropout during pre-training, and C4 only. The **LM-adapted** T5.1.1 checkpoints were trained for a further 100K steps with a plain language-modelling objective. They were used for prompt tuning (Lester et al., 2021) and as the starting point of T0.

**BART** pairs a BERT-style encoder with a GPT-style decoder and learns to reconstruct documents from text infilling and sentence shuffling. mBART (Liu et al., 2020) applied the same objective to 25 languages for translation. **mT5** (Xue et al., "mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer", NAACL 2021) trained T5.1.1 on mC4, covering 101 languages, with a 250K SentencePiece vocabulary, in sizes from 300M to 13B. ByT5 (Xue et al., TACL 2022) dropped the tokenizer altogether, read UTF-8 bytes and made the encoder three times deeper than the decoder.

**T0** (Sanh et al., ICLR 2022) and **Flan-T5** (Chung et al., "Scaling Instruction-Finetuned Language Models", 2022) showed that instruction tuning turns T5 into a zero-shot model. Flan-T5 was fine-tuned on a collection of 1,836 tasks, many phrased with chain-of-thought or few-shot templates. UL2 (Tay et al., ICLR 2023) pre-trained a 20B encoder–decoder on a mixture of denoisers. **Whisper** (Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision", 2022) is the clearest modern case. Its encoder reads 30 seconds of 80-channel log-Mel spectrogram, downsampled by two convolution layers to 1,500 positions. Its decoder writes the transcript, with special tokens choosing the language and the task (transcribe or translate). It was trained on 680,000 hours of labelled audio.

**Configurations from the papers**

- **T5-11B:** 24 encoder + 24 decoder layers, $d_{\text{model}}=1024$, $d_{ff}=65536$, 128 heads with $d_{kv}=128$; the 3B model uses $d_{ff}=16384$ and 32 heads
- **T5 pre-training:** $2^{19}=524{,}288$ steps, batch 128 sequences of 512 tokens ($2^{16}$ tokens), about 34B tokens; 32K SentencePiece vocabulary; relative position biases with 32 buckets
- **BART-large:** 12 + 12 layers, d = 1024, about 400M parameters, pre-trained on 160 GB of text with RoBERTa's batch size and data
- **mT5:** mC4, 101 languages, 250K-token vocabulary, Small 300M to XXL 13B; at small sizes the embeddings are most of the parameters (mT5-Small 300M against T5-Small 60M)
- **Flan-T5:** T5 checkpoints from 80M to 11B, instruction-tuned on 1,836 tasks
- **Whisper large:** 32 + 32 layers, width 1280, 20 heads, 1.55B parameters; encoder input 3,000 Mel frames → 1,500 positions; decoder context 448 tokens; sizes 39M (tiny) to 1.55B

## Compute per token, compared with decoder-only

Take an encoder–decoder with $L$ layers per stack and width $d$, and set $d_{ff}=4d$. An encoder layer has $12d^2$ weights. A decoder layer has $16d^2$, since cross-attention adds four more $d\times d$ projections. Each input token passes only through the encoder and each output token only through the decoder. So the model holds $28Ld^2$ parameters, yet a token meets only $12d^2$ to $14d^2$ of them per layer. Counting 2 FLOPs per multiply–add and $4d$ per attended query–key pair:

$$
F_{\text{ED}}=L\big[\underbrace{24d^2I+4dI^2}_{\text{encoder}}+\underbrace{4d^2I}_{\text{cross }K,V}\big]+L\big[\underbrace{28d^2O}_{\text{decoder weights}}+\underbrace{2dO(O+1)}_{\text{causal self}}+\underbrace{4dIO}_{\text{cross}}\big],\qquad F_{\text{D}}=L_D\big[24d^2(I+O)+2d\,(I+O)(I+O+1)\big]
$$

- $24d^2$: $2\times12d^2$: the encoder layer's weights applied to one token; the decoder applies $2\times14d^2$ (cross $K,V$ projections act on $m$ instead)
- $4dI^2$: bidirectional encoder attention over all $I^2$ pairs: **twice** the $2dI(I+1)$ a causal prefix pays
- $L_D$: decoder-only depth; $L_D=\tfrac{28}{12}L$ matches parameters, $L_D=L$ matches the depth each token sees
- $\text{KV}$: cache: $2Ld(I+O)$ numbers for the encoder–decoder (cross + self) against $2L_Dd(I+O)$

Two comparisons follow. Take first a decoder-only model with the **same parameters** ($L_D=56$ for $L=24$). It needs 1.3–2.1 times the FLOPs of the encoder–decoder (about 2× unless the input is very long), a cache $56/24=2.33$ times larger, and twice the weights read per generated token. Decoding is memory-bound, so that last factor is close to a 2× speed-up for the encoder–decoder. Against a decoder with the **same depth per token** ($L_D=24$), the encoder–decoder has 2.33 times the parameters for 1.1–1.8 times the FLOPs. The upper end comes from long inputs, where full encoder attention costs twice what a causal prefix pays.

This is why Wang et al. ("What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?", ICML 2022) compared architectures at matched compute rather than matched size. Their encoder–decoders had about twice the parameters of their decoder-only models. Under that matching, the causal decoder with a plain LM objective was best at zero-shot use straight after pre-training. Encoder–decoder and non-causal decoders trained with masked denoising and then multitask-finetuned were best overall. Pre-training objective and architecture interact: an encoder–decoder pre-trained with span corruption needs multitask or instruction tuning before it can be prompted, which is exactly what T0 and Flan-T5 add.

> Figure: Forward FLOPs, KV cache, weights read per generated token and parameters for one example. The models are an encoder–decoder with 24 + 24 layers (T5-Large shape, 705M non-embedding parameters), a decoder-only model with 56 layers (the same 705M), and one with 24 layers (302M, the same depth per token), all at d = 1024. At the default summarisation setting, 4,096 → 256, the encoder–decoder needs 4.82 TFLOP and a 408 MiB cache. The same-size decoder needs 8.31 TFLOP (1.72×) and 952 MiB (2.33×), and reads 705M rather than 352M weights per token (2.00×). The 24-layer decoder needs 3.56 TFLOP (0.74×), so the encoder–decoder's 2.33× parameters cost 1.35× compute. Encoder attention is 34.2% of its FLOPs. For translation (64 → 64) the ratios are 2.00× and 0.86×, and encoder attention is 0.4%. For long-document QA (32K → 64) they are 1.32× and 0.56×, and encoder attention is 81.9%: full attention over 32K tokens is the price of reading them bidirectionally. For a chat reply (256 → 4K) the ratios are 2.08× and 0.89×. Most of the work is in the decoder, and the split buys little.

## When the encoder–decoder wins

**The input is given and much longer than the output.** Summarisation, document QA, classification phrased as generation, and retrieval-augmented reading all fit. The encoder reads the input once, bidirectionally, and every output step then reads only the decoder half of the weights. Fusion-in-Decoder (Izacard and Grave, EACL 2021) pushes this further. It encodes each retrieved passage **independently** with its question, concatenates the memories, and lets the decoder cross-attend to all of them. Encoder cost then grows linearly with the number of passages rather than quadratically with their total length.

**Input and output are different kinds of sequence.** In translation the source language is read and the target language is written, so separate weights for reading and writing are a natural inductive bias. In speech recognition the input is not text at all. Whisper's encoder handles 1,500 audio positions, its decoder a few hundred text tokens. The modality split matches the architectural split. The same holds for image captioning and OCR models with vision encoders.

**It loses** when the boundary between input and output is fluid. In multi-turn chat, each reply becomes part of the next input. An encoder–decoder must then re-encode the whole history, because new tokens change every bidirectional representation, while a decoder-only model just extends its **KV cache**. In-context learning, open-ended continuation and agentic loops follow the same pattern. Ecosystem effects compound the gap: nearly all frontier-scale training, tooling and inference work since 2021 has targeted decoder-only models.

## Worked example: one summarisation example, counted

Take a small encoder–decoder with $L=6$ layers per stack and $d=512$ (the shape of Transformer base), an input of $I=1000$ tokens and an output of $O=100$. Compare it with a 14-layer decoder-only model, which has the same $28\cdot6=168$ units of $d^2$ in parameters, and with a 6-layer one.

Forward FLOPs term by term (computed in Python; 1 GFLOP = 10⁹)

| term | formula | GFLOP |
| --- | --- | --- |
| encoder weights | L · 24d² · I | 37.75 |
| encoder attention | L · 4d · I² | 12.29 |
| cross K, V of the memory | L · 4d² · I | 6.29 |
| decoder weights | L · 28d² · O | 4.40 |
| decoder causal self-attention | L · 2d · O(O + 1) | 0.06 |
| cross-attention | L · 4d · I · O | 1.23 |
| encoder–decoder total |  | 62.02 |
| decoder-only, 14 layers (same parameters) | 14 · [24d² · 1100 + 2d · 1100 · 1101] | 114.25 |
| decoder-only, 6 layers (same depth) | 6 · [24d² · 1100 + 2d · 1100 · 1101] | 48.96 |

**Reading the numbers**

1. Parameters: encoder–decoder $28\cdot6\cdot512^2=44{,}040{,}192$, the same as the 14-layer decoder ($14\cdot12\cdot512^2$). The 6-layer decoder has $18{,}874{,}368$.
2. The input dominates: the encoder side ($37.75+12.29+6.29=56.33$ GFLOP) is 91% of the total, because $I=10\,O$.
3. Against the same-size decoder: $114.25/62.02=1.84$ times the FLOPs for the decoder-only model. Against the same-depth decoder: the encoder–decoder costs $62.02/48.96=1.27$ times as much, for $44.0/18.9=2.33$ times the parameters.
4. KV cache: encoder–decoder $2\cdot6\cdot512\cdot(1000+100)=6{,}758{,}400$ numbers (13.5 MB in bf16), the same as the 6-layer decoder. The 14-layer decoder needs $15{,}769{,}600$ (2.33×).
5. Weights read per generated token: $14\cdot6\cdot512^2=22.0$M for the encoder–decoder against $44.0$M for the same-size decoder. Memory-bound decoding is about twice as fast.

```python
def flops_encdec(I, O, L, d):
    """Forward FLOPs for one example: encoder over I tokens, decoder over O tokens."""
    encode = L * (24 * d * d * I + 4 * d * I * I)      # 12d² weights, all I² attention pairs
    cross_kv = L * 4 * d * d * I                       # K, V of the memory, once per input
    decode = L * (28 * d * d * O + 2 * d * O * (O + 1) + 4 * d * I * O)
    return encode + cross_kv + decode

def flops_decoder(I, O, L, d):
    T = I + O                                          # prompt and answer in one causal stream
    return L * (24 * d * d * T + 2 * d * T * (T + 1))

d, L = 1024, 24                                        # T5-Large shape: 24 + 24 layers
for I, O in [(64, 64), (4096, 256), (32768, 64), (256, 4096)]:
    ed = flops_encdec(I, O, L, d)
    same_params = flops_decoder(I, O, 56, d)           # 56 · 12d² = 24 · (12 + 16) d²
    same_depth = flops_decoder(I, O, 24, d)
    print(f"{I:>6} -> {O:<5} enc-dec {ed / 1e12:6.2f} TFLOP | "
          f"dec-56L {same_params / ed:.2f}x | dec-24L {same_depth / ed:.2f}x")
#     64 -> 64    enc-dec   0.09 TFLOP | dec-56L 2.00x | dec-24L 0.86x
#   4096 -> 256   enc-dec   4.82 TFLOP | dec-56L 1.72x | dec-24L 0.74x
#  32768 -> 64    enc-dec 128.89 TFLOP | dec-56L 1.32x | dec-24L 0.56x
#    256 -> 4096  enc-dec   4.00 TFLOP | dec-56L 2.08x | dec-24L 0.89x

# --- Using real encoder–decoders with Hugging Face transformers ---
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline

tok = AutoTokenizer.from_pretrained("google/flan-t5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base").eval()

doc = "summarize: " + open("report.txt").read()
enc_in = tok(doc, return_tensors="pt", truncation=True, max_length=2048)
with torch.no_grad():
    memory = model.get_encoder()(**enc_in)             # the encoder runs once: m has shape (1, I, 768)
    greedy = model.generate(encoder_outputs=memory, attention_mask=enc_in.attention_mask, max_new_tokens=80)
    sampled = model.generate(encoder_outputs=memory, attention_mask=enc_in.attention_mask,
                             max_new_tokens=80, do_sample=True, top_p=0.9)
print(tok.batch_decode(greedy, skip_special_tokens=True)[0])

# Training: labels are the target; the decoder input is the labels shifted right internally.
labels = tok("A two-sentence summary.", return_tensors="pt").input_ids
loss = model(**enc_in, labels=labels).loss              # cross-entropy over the O target tokens only

# Whisper: 30 s of audio -> log-Mel -> 1,500 encoder positions -> text tokens.
asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
print(asr("meeting.wav", return_timestamps=True)["text"])
```

The FLOP model of the figure (plain Python, run; output in comments), then Flan-T5 with the encoder run once and two decodes reusing it, and Whisper through the pipeline API (transformers; not run here)

> **Parameters and compute come apart:** An encoder–decoder is the oldest way to have more parameters than any one token uses: input tokens use the encoder, output tokens the decoder. In the figure it holds 2.33 times the parameters of a same-depth decoder for 1.1–1.8 times the FLOPs. **Mixture of experts (MoE)** takes the same idea much further by routing tokens through a few of many expert MLPs.

> **Appending to the input invalidates the encoder:** The memory $m$ depends on every input token, because encoder attention is bidirectional. Add one sentence to the document, or a new turn to a conversation, and the whole encoder pass must be redone. Cross-attention keys and values of the old memory cannot be reused. Workloads with growing context (chat, agents, streaming) lose the encoder–decoder's efficiency advantage. Whisper avoids the problem by processing audio in fixed 30-second windows.

> **Trained lengths are short:** T5 was pre-trained with 512-token inputs and Whisper with 30-second windows and 448 decoder tokens. T5's relative position buckets let it accept longer inputs, but quality falls off beyond the lengths seen in training, and encoder attention memory grows as $I^2$. Long-input encoder–decoders such as LongT5 and the BigBird and Longformer encoder–decoders replace the full encoder attention with sparse or local patterns (see **Longformer**, **BigBird**).

**Strengths**

- Bidirectional reading of the input: every input token sees the whole input in every layer
- About 2× the parameters of a decoder at similar per-token compute, and about half the weights read per generated token at equal size
- Smaller KV cache at equal size; cross-attention keys and values are computed once per input
- A natural fit for conditional tasks and modality changes: translation, summarisation, speech recognition, captioning
- Fusion-in-Decoder style reading of many passages at linear encoder cost

**Limitations**

- Needs a fixed split into input and output; growing inputs (chat, agents) force a full re-encode
- Denoising-pretrained checkpoints need multitask or instruction tuning before zero-shot prompting works
- Full encoder attention costs twice a causal prefix at long inputs
- Two stacks, three attention types and a smaller ecosystem than decoder-only models at frontier scale

**Question:** An encoder–decoder with 24 + 24 layers and a decoder-only model with 56 layers have the same parameter count. When generating a 256-token summary of a 4,096-token document, why is the encoder–decoder faster per generated token?

- A. It uses a smaller vocabulary
- B. Each decode step runs only the 24 decoder layers (about 14d² weights each, with cross K, V precomputed), so it reads half the weights; its KV cache is also 2.33× smaller
- C. Cross-attention is cheaper than self-attention per pair
- D. It generates several tokens per step

**Answer:** B. Each decode step runs only the 24 decoder layers (about 14d² weights each, with cross K, V precomputed), so it reads half the weights; its KV cache is also 2.33× smaller — Per step the encoder–decoder touches $24\cdot14d^2\approx352$M weights against $56\cdot12d^2\approx705$M. Memory-bound decoding scales with bytes read, so that is about a 2× speed-up. The input was paid for once, in the encoder pass.

**Question:** Fusion-in-Decoder answers a question from 100 retrieved passages of 200 tokens each. Why does it encode the passages independently rather than as one 20,000-token input?

- A. Independent encoding lets the decoder skip cross-attention
- B. Encoder attention is quadratic in the encoded length: 100 separate 200-token passes cost 100 × 200² pairs instead of 20,000², and the decoder still cross-attends to all 20,000 memory vectors
- C. T5 cannot read inputs longer than 512 tokens under any circumstances
- D. It reduces the number of decoder layers

**Answer:** B. Encoder attention is quadratic in the encoded length: 100 separate 200-token passes cost 100 × 200² pairs instead of 20,000², and the decoder still cross-attends to all 20,000 memory vectors — $100\cdot200^2=4\cdot10^6$ attention pairs against $(2\cdot10^4)^2=4\cdot10^8$, which is 100 times fewer. Passages cannot attend to each other in the encoder, but the decoder fuses them through cross-attention over the concatenated memory.

**Key points**

- Encoder–decoder LMs model $p(y\mid x)=\prod_t p(y_t\mid y_{<t},\mathrm{Enc}(x))$: a bidirectional encoder run once, a causal decoder with cross-attention, and a loss only on the output.
- Family: T5 (C4, span corruption, 60M–11B), BART (denoising), mT5 (101 languages), Flan-T5 (1,836 instruction tasks), UL2, Whisper (680K hours of speech), NLLB-200 (translation).
- Each token crosses one stack: $28Ld^2$ parameters, but only $12$–$14\,d^2$ per layer per token. That is 2.33× the parameters of a same-depth decoder for 1.1–1.8× its FLOPs, or half the weights per decode step at equal size.
- It wins when the input is fixed and long relative to the output, or in a different modality. It loses when the input keeps growing, as in chat and agents, which forces a re-encode.
- Wang et al. (2022): causal decoders are best zero-shot straight after pre-training; non-causal models with denoising are best after multitask finetuning.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
