Encoder–decoder architecture

An encoder reads the input into a representation and a decoder generates the output from it, one token at a time, as a conditional language model. Squeezing the input into one fixed vector is the weak point; attention and skip connections remove that bottleneck.

12 min read1262 wordscore0 viewsupdated 2026-10-12

Definition

Encoder–decoder architecture is a network made of two parts. The encoder maps an input xx (a sentence, an image, an audio clip) to an intermediate representation c=enc(x)c=\mathrm{enc}(x). The decoder produces the output yy from that representation, usually autoregressively: p(y∣x)=∏tp(yt∣y<t,c)p(y\mid x)=\prod_t p(y_t\mid y_{<t},c). Input and output can have different lengths, different vocabularies and even different modalities.

Many tasks map one structured object to another of a different shape: translation (a French sentence of length 7 to an English sentence of length 9), speech recognition (audio frames to characters), captioning (an image to words), summarisation, segmentation (an image to a per-pixel mask). A single recurrent network that emits one output per input step cannot handle unequal lengths or reordering. Splitting the job in two is the fix: first understand the whole input, then generate the output conditioned on that understanding.

The pattern is old in the form of the autoencoder, where the decoder reconstructs the encoder's own input (Hinton and Salakhutdinov, 2006). For sequence transduction it arrived in 2013–2014. Kalchbrenner and Blunsom (2013) encoded a source sentence with a convolutional model and decoded it with a recurrent language model. Cho, van Merriënboer, Gulcehre, Bahdanau, Bougares, Schwenk and Bengio (EMNLP 2014) named the RNN Encoder–Decoder: one RNN reads the source into a fixed-length vector, a second RNN generates the target from it (the same paper introduced the GRU). Sutskever, Vinyals and Le (2014) scaled the idea up with deep LSTMs (see Sequence-to-sequence (Seq2Seq)). The same two-part design then spread to images (Show and Tell, U-Net), and the Transformer (Vaswani et al., 2017) is an encoder–decoder too.

The conditional language model

c=enc(x1,…,xT),p(y1,…,yT′∣x)=∏t=1T′p(yt∣y1,…,yt−1, c)c=\mathrm{enc}(x_1,\dots,x_T),\qquad p(y_1,\dots,y_{T'}\mid x)=\prod_{t=1}^{T'} p\big(y_t\mid y_1,\dots,y_{t-1},\,c\big)
x1..xTx_1..x_T
the input sequence (or image, or audio), length TT
cc
the encoder's representation: one fixed vector (classic RNN encoder–decoder) or one vector per input position (attention, Transformer)
y1..yT′y_1..y_{T'}
the output sequence, length T′T' chosen by the decoder itself: it stops when it emits an end-of-sequence token
p(yt∣y<t,c)p(y_t\mid y_{<t},c)
the decoder: a language model over the output vocabulary whose every step is conditioned on cc

The factorisation is exact (it is the chain rule of probability), so the decoder is just a language model with an extra input. Training maximises log⁡p(y∣x)=∑tlog⁡p(yt∣y<t,c)\log p(y\mid x)=\sum_t\log p(y_t\mid y_{<t},c) over (input, output) pairs: a cross-entropy loss per output token, with the ground-truth prefix y<ty_{<t} fed in (see Teacher forcing). The gradient flows from every output token back through the decoder into the encoder, so both halves are trained jointly, end to end, for the one task. At inference the decoder generates token by token, by greedy choice, sampling, or Beam search.

In the RNN version the encoder's final hidden state is the representation and becomes the decoder's initial state: ht=fenc(ht−1,xt)h_t=f_{\text{enc}}(h_{t-1},x_t), c=hTc=h_T, s0=cs_0=c (or s0=tanh⁡(Wc)s_0=\tanh(Wc)), st=fdec(st−1,yt−1,c)s_t=f_{\text{dec}}(s_{t-1},y_{t-1},c), p(yt∣⋅)=softmax(Wost)p(y_t\mid\cdot)=\mathrm{softmax}(W_o s_t). Cho et al. fed cc into every decoder step as well, so it would not fade from the decoder state.

At a glance

Encoder
any network that reads the whole input: RNN/LSTM/GRU, CNN, Transformer encoder (bidirectional, sees everything)
Decoder
a conditional generator, usually autoregressive and causal: sees only y<ty_{<t} and the encoder output
Interface
one fixed vector (RNN encoder–decoder, Show and Tell), or all encoder states read by attention (Bahdanau, Transformer cross-attention), or skip connections (U-Net)
Loss
−∑tlog⁡p(yt∣y<t,x)-\sum_t\log p(y_t\mid y_{<t},x), summed over output tokens; trained end to end
Lengths
TT and T′T' are independent; the decoder ends with an <eos> token
Weak point
a fixed-size cc must hold the whole input: quality drops on long inputs (Cho et al., 2014b)

Across modalities

systemencoderrepresentationdecoder
RNN Encoder–Decoder (Cho et al., 2014)GRU over source wordsfinal hidden state cc, fed to every decoder stepGRU language model over target words
Seq2Seq (Sutskever et al., 2014)4-layer LSTM over the reversed sourcefinal (h,c)(h,c) of all layers4-layer LSTM, initialised from the encoder
RNNsearch (Bahdanau et al., 2015)bidirectional GRUone annotation per source word, read by attentionGRU with a new context vector each step
Show and Tell (Vinyals et al., 2015)ImageNet CNNimage embedding, given to the LSTM once, before the first wordLSTM caption generator
U-Net (Ronneberger et al., 2015)contracting conv pathbottleneck feature map plus skip connections at every scaleexpanding path with up-convolutions to a per-pixel map
AutoencoderMLP or CNNlow-dimensional code zzreconstructs the input xx
Transformer, T5, BART, Whisperstack of self-attention layersone vector per input tokencausal self-attention plus cross-attention to every encoder vector
The same two-part pattern in different systems

Two lessons sit in the table. First, the encoder and the decoder need not be the same kind of network: Show and Tell (Vinyals, Toshev, Bengio and Erhan, CVPR 2015) pairs a convolutional image encoder with an LSTM text decoder, and Whisper (Radford et al., 2022) pairs an audio Transformer encoder with a text decoder. Second, the systems that win on long or detailed inputs all pass more than one vector across: attention reads every encoder state, U-Net copies feature maps across at every resolution. T5 (Raffel et al., JMLR 2020) cast every NLP task as text-to-text with an encoder–decoder Transformer, and in its comparison of architectures the encoder–decoder with a denoising objective did best.

The bottleneck of a fixed vector

If the whole input must pass through a vector of dd numbers, there is a hard limit on what the decoder can know, and it gets worse as the input grows. Cho, van Merriënboer, Bahdanau and Bengio ("On the Properties of Neural Machine Translation: Encoder–Decoder Approaches", SSST 2014) measured it: the BLEU score of their RNN encoder–decoder fell steadily as sentences got longer, and they named the fixed-length vector as the likely cause. Bahdanau, Cho and Bengio's attention model, published the same year, was the answer: keep all TT encoder states and let the decoder choose which to read at each step.

The figure makes the limit exact with a linear model. A linear RNN with a dd-dimensional state, ht=ρQht−1+b xth_t=\rho Q h_{t-1}+b\,x_t (QQ a random rotation, bb a random unit vector), reads TT random numbers xt∼N(0,1)x_t\sim\mathcal N(0,1). Its final state is c=Mxc=Mx for a d×Td\times T matrix MM. The decoder is the best possible linear decoder when every number it receives has noise of standard deviation σ=0.05\sigma=0.05. Its recovered share of token tt's variance is rt∈[0,1]r_t\in[0,1], and because MM has rank at most dd,

∑t=1Trt  ≤  d⟹R2‾=1T∑trt  ≤  min⁡ ⁣(1,dT)\sum_{t=1}^{T} r_t\;\le\;d\qquad\Longrightarrow\qquad \overline{R^2}=\frac1T\sum_t r_t\;\le\;\min\!\Big(1,\frac dT\Big)
rtr_t
[(M⊤M+σ2I)−1M⊤M]tt[(M^\top M+\sigma^2I)^{-1}M^\top M]_{tt}, the fraction of Var⁡(xt)\operatorname{Var}(x_t) the best linear decoder recovers from cc
dd
the size of the fixed vector
R2‾\overline{R^2}
mean recovered variance over the sequence
source x₁..x_T (bars) and decoder reconstruction x̂ (dots) · T = 20−22decoder input: c = h_T, 8 numbers for 20 tokens2.1−1.50.90.82.61.62.62.7share of each token recovered, r_tmean R² 0.399 · Σ r_t = 7.99 ≤ d = 810t = 1t = 20 (read last)mean R² vs length T140┄ bound min(1, d/T) = 0.40■ xₜ ● x̂ₜ · encoder hₜ = ρQhₜ₋₁ + bxₜ · best linear decoder, σ = 0.05 · MSE 0.820
decoder reads
fig.A linear RNN encoder squeezes T random numbers into a vector of d = 8 numbers; the best linear decoder reconstructs them (orange dots over the blue source bars). The bottom-left bars show how much of each token survives and the right curve shows mean R² against T, with the bound min(1, d/T) dashed. With ρ = 1, mean R² is 0.998 at T = 1, 0.952 at T = 8, 0.798 at T = 10, 0.399 at T = 20 and 0.200 at T = 40: once T > d it tracks d/T, and the 8 numbers never carry more than 8 tokens' worth (Σ r_t = 7.99 at T = 20). Lower ρ makes the encoder forget: at ρ = 0.8, T = 20 the first token keeps 0.3% of its variance and the last 96.7%. Switch the decoder to read all encoder states, as attention does, and mean R² is 0.999 at every length.

The curve has the shape Cho et al. saw in BLEU: fine while the input fits, then a steady decline. A nonlinear encoder can pack information more cleverly than a linear one, but no encoder can beat the counting argument in general: dd real numbers at finite precision hold a bounded number of bits, while the input's information grows with TT. The forgetting factor ρ\rho adds a second effect. A recurrent encoder overwrites old information with new, so the first tokens are the worst preserved, exactly the ones a left-to-right decoder needs first. That is why Sutskever et al. reversed the source sentence, and why attention, which skips the bottleneck entirely, made the trick unnecessary.

Worked example: one number for two tokens

Take d=1d=1, T=2T=2, ρ=1\rho=1. In one dimension a rotation is ±1\pm1, and the figure's encoder gives M=(−1, −1)M=(-1,\,-1), so c=−(x1+x2)c=-(x_1+x_2). With noise variance σ2=0.0025\sigma^2=0.0025:

  1. 1M⊤M=(1111)M^\top M=\begin{pmatrix}1&1\\1&1\end{pmatrix}. Its eigenvalues are 2 (direction (1,1)(1,1)) and 0 (direction (1,−1)(1,-1)).
  2. 2The decoder recovers the sum and nothing of the difference: (M⊤M+σ2I)−1M⊤M=12+σ2(1111)(M^\top M+\sigma^2I)^{-1}M^\top M=\frac{1}{2+\sigma^2}\begin{pmatrix}1&1\\1&1\end{pmatrix}.
  3. 3So r1=r2=1/(2+0.0025)=0.4994r_1=r_2=1/(2+0.0025)=0.4994 and x^1=x^2=x1+x22.0025\hat x_1=\hat x_2=\tfrac{x_1+x_2}{2.0025}: each token is half recovered.
  4. 4∑trt=0.9988≤d=1\sum_t r_t=0.9988\le d=1, mean R2=0.4994≤d/T=0.5R^2=0.4994\le d/T=0.5. The figure's library returns the same 0.4994.
  5. 5Give the decoder both states h1=−x1h_1=-x_1 and h2=−(x1+x2)h_2=-(x_1+x_2) instead, and x1=−h1x_1=-h_1, x2=h1−h2x_2=h_1-h_2: the bottleneck is gone.

The output side of the same model is the chain-rule factorisation. If a trained decoder assigns the three reference tokens of a translation the probabilities 0.6, 0.5 and 0.9 given their prefixes, then p(y∣x)=0.6⋅0.5⋅0.9=0.27p(y\mid x)=0.6\cdot0.5\cdot0.9=0.27 and the training loss for this pair is −ln⁡0.6−ln⁡0.5−ln⁡0.9=0.511+0.693+0.105=1.309-\ln 0.6-\ln 0.5-\ln 0.9=0.511+0.693+0.105=1.309 nats, or 0.436 per token.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class EncoderDecoder(nn.Module):
4 """Cho et al. (2014): c = last encoder state, fed to every decoder step."""
5 def __init__(self, src_vocab, tgt_vocab, emb=256, hid=512):
6 super().__init__()
7 self.src_emb = nn.Embedding(src_vocab, emb)
8 self.tgt_emb = nn.Embedding(tgt_vocab, emb)
9 self.encoder = nn.GRU(emb, hid, batch_first=True)
10 self.init = nn.Linear(hid, hid)
11 self.decoder = nn.GRU(emb + hid, hid, batch_first=True) # input = [y_{t-1}; c]
12 self.out = nn.Linear(hid + hid, tgt_vocab) # logits from [s_t; c]
13
14 def encode(self, src): # src: (N, T) token ids
15 _, h_n = self.encoder(self.src_emb(src)) # h_n: (1, N, hid)
16 return h_n[-1] # c: (N, hid), the whole input in one vector
17
18 def forward(self, src, tgt_in): # tgt_in: (N, T') = <sos> y_1 .. y_{T'-1}
19 c = self.encode(src)
20 s0 = torch.tanh(self.init(c)).unsqueeze(0)
21 ctx = c.unsqueeze(1).expand(-1, tgt_in.size(1), -1)
22 s, _ = self.decoder(torch.cat([self.tgt_emb(tgt_in), ctx], dim=-1), s0)
23 return self.out(torch.cat([s, ctx], dim=-1)) # (N, T', tgt_vocab)
24
25model = EncoderDecoder(src_vocab=8000, tgt_vocab=8000)
26src = torch.randint(0, 8000, (32, 17))
27tgt = torch.randint(0, 8000, (32, 12)) # <sos> ... <eos>
28logits = model(src, tgt[:, :-1]) # teacher forcing: shifted ground truth
29loss = F.cross_entropy(logits.reshape(-1, 8000), tgt[:, 1:].reshape(-1))
30
31# The Transformer is the same split, but the decoder cross-attends to ALL encoder vectors.
32tf = nn.Transformer(d_model=512, nhead=8, num_encoder_layers=6,
33 num_decoder_layers=6, batch_first=True)
34x, y = torch.randn(32, 17, 512), torch.randn(32, 11, 512) # embedded source / shifted target
35memory = tf.encoder(x) # (32, 17, 512): one vector per token
36mask = nn.Transformer.generate_square_subsequent_mask(11) # causal mask for the decoder
37h = tf.decoder(y, memory, tgt_mask=mask) # (32, 11, 512)
A GRU encoder–decoder with a fixed context vector, and the same pattern as nn.Transformer

Use when

  • +Input and output lengths, vocabularies and even modalities are decoupled
  • +One principled loss: the conditional log-likelihood ∑tlog⁡p(yt∣y<t,x)\sum_t\log p(y_t\mid y_{<t},x), trained end to end
  • +The encoder sees the whole input (bidirectionally) before any output is produced
  • +Encoder and decoder can be swapped or pre-trained separately (CNN encoder + LSTM decoder, pretrained T5 encoder)

Avoid when

  • −A single fixed vector caps what is transmitted: quality falls with input length
  • −Autoregressive decoding is sequential: output latency grows with output length
  • −Training with ground-truth prefixes but generating from its own outputs causes exposure bias
  • −Two networks to size and tune; the decoder may learn to ignore a weak encoder

Check yourself

An RNN encoder–decoder passes a single 256-dimensional vector from encoder to decoder. Translation quality is good for 10-word sentences but poor for 50-word ones. Which change addresses the cause most directly?

Check yourself

In the figure's linear model with d = 4, what is the largest possible mean R² for a sequence of T = 16 tokens?

Remember

  • ✓Encoder: input →\to representation cc. Decoder: a conditional language model p(y∣x)=∏tp(yt∣y<t,c)p(y\mid x)=\prod_t p(y_t\mid y_{<t},c).
  • ✓Trained end to end with per-token cross-entropy; generation is token by token until an end-of-sequence token.
  • ✓The pattern spans modalities: RNN Seq2Seq, Show and Tell captioning, U-Net, autoencoders, Transformer, T5, Whisper.
  • ✓A fixed vector of dd numbers can carry at most dd tokens' worth of a linear code, so quality falls with input length (Cho et al., 2014).
  • ✓Attention and skip connections widen the interface to one vector per input position and remove the bottleneck.
← back to Deep Learning roadmap