Definition
Encoder–decoder architecture is a network made of two parts. The encoder maps an input (a sentence, an image, an audio clip) to an intermediate representation . The decoder produces the output from that representation, usually autoregressively: . Input and output can have different lengths, different vocabularies and even different modalities.
Many tasks map one structured object to another of a different shape: translation (a French sentence of length 7 to an English sentence of length 9), speech recognition (audio frames to characters), captioning (an image to words), summarisation, segmentation (an image to a per-pixel mask). A single recurrent network that emits one output per input step cannot handle unequal lengths or reordering. Splitting the job in two is the fix: first understand the whole input, then generate the output conditioned on that understanding.
The pattern is old in the form of the autoencoder, where the decoder reconstructs the encoder's own input (Hinton and Salakhutdinov, 2006). For sequence transduction it arrived in 2013–2014. Kalchbrenner and Blunsom (2013) encoded a source sentence with a convolutional model and decoded it with a recurrent language model. Cho, van Merriënboer, Gulcehre, Bahdanau, Bougares, Schwenk and Bengio (EMNLP 2014) named the RNN Encoder–Decoder: one RNN reads the source into a fixed-length vector, a second RNN generates the target from it (the same paper introduced the GRU). Sutskever, Vinyals and Le (2014) scaled the idea up with deep LSTMs (see Sequence-to-sequence (Seq2Seq)). The same two-part design then spread to images (Show and Tell, U-Net), and the Transformer (Vaswani et al., 2017) is an encoder–decoder too.
The conditional language model
- the input sequence (or image, or audio), length
- the encoder's representation: one fixed vector (classic RNN encoder–decoder) or one vector per input position (attention, Transformer)
- the output sequence, length chosen by the decoder itself: it stops when it emits an end-of-sequence token
- the decoder: a language model over the output vocabulary whose every step is conditioned on
The factorisation is exact (it is the chain rule of probability), so the decoder is just a language model with an extra input. Training maximises over (input, output) pairs: a cross-entropy loss per output token, with the ground-truth prefix fed in (see Teacher forcing). The gradient flows from every output token back through the decoder into the encoder, so both halves are trained jointly, end to end, for the one task. At inference the decoder generates token by token, by greedy choice, sampling, or Beam search.
In the RNN version the encoder's final hidden state is the representation and becomes the decoder's initial state: , , (or ), , . Cho et al. fed into every decoder step as well, so it would not fade from the decoder state.
At a glance
- Encoder
- any network that reads the whole input: RNN/LSTM/GRU, CNN, Transformer encoder (bidirectional, sees everything)
- Decoder
- a conditional generator, usually autoregressive and causal: sees only and the encoder output
- Interface
- one fixed vector (RNN encoder–decoder, Show and Tell), or all encoder states read by attention (Bahdanau, Transformer cross-attention), or skip connections (U-Net)
- Loss
- , summed over output tokens; trained end to end
- Lengths
- and are independent; the decoder ends with an
<eos>token - Weak point
- a fixed-size must hold the whole input: quality drops on long inputs (Cho et al., 2014b)
Across modalities
| system | encoder | representation | decoder |
|---|---|---|---|
| RNN Encoder–Decoder (Cho et al., 2014) | GRU over source words | final hidden state , fed to every decoder step | GRU language model over target words |
| Seq2Seq (Sutskever et al., 2014) | 4-layer LSTM over the reversed source | final of all layers | 4-layer LSTM, initialised from the encoder |
| RNNsearch (Bahdanau et al., 2015) | bidirectional GRU | one annotation per source word, read by attention | GRU with a new context vector each step |
| Show and Tell (Vinyals et al., 2015) | ImageNet CNN | image embedding, given to the LSTM once, before the first word | LSTM caption generator |
| U-Net (Ronneberger et al., 2015) | contracting conv path | bottleneck feature map plus skip connections at every scale | expanding path with up-convolutions to a per-pixel map |
| Autoencoder | MLP or CNN | low-dimensional code | reconstructs the input |
| Transformer, T5, BART, Whisper | stack of self-attention layers | one vector per input token | causal self-attention plus cross-attention to every encoder vector |
Two lessons sit in the table. First, the encoder and the decoder need not be the same kind of network: Show and Tell (Vinyals, Toshev, Bengio and Erhan, CVPR 2015) pairs a convolutional image encoder with an LSTM text decoder, and Whisper (Radford et al., 2022) pairs an audio Transformer encoder with a text decoder. Second, the systems that win on long or detailed inputs all pass more than one vector across: attention reads every encoder state, U-Net copies feature maps across at every resolution. T5 (Raffel et al., JMLR 2020) cast every NLP task as text-to-text with an encoder–decoder Transformer, and in its comparison of architectures the encoder–decoder with a denoising objective did best.
The bottleneck of a fixed vector
If the whole input must pass through a vector of numbers, there is a hard limit on what the decoder can know, and it gets worse as the input grows. Cho, van Merriënboer, Bahdanau and Bengio ("On the Properties of Neural Machine Translation: Encoder–Decoder Approaches", SSST 2014) measured it: the BLEU score of their RNN encoder–decoder fell steadily as sentences got longer, and they named the fixed-length vector as the likely cause. Bahdanau, Cho and Bengio's attention model, published the same year, was the answer: keep all encoder states and let the decoder choose which to read at each step.
The figure makes the limit exact with a linear model. A linear RNN with a -dimensional state, ( a random rotation, a random unit vector), reads random numbers . Its final state is for a matrix . The decoder is the best possible linear decoder when every number it receives has noise of standard deviation . Its recovered share of token 's variance is , and because has rank at most ,
- , the fraction of the best linear decoder recovers from
- the size of the fixed vector
- mean recovered variance over the sequence
The curve has the shape Cho et al. saw in BLEU: fine while the input fits, then a steady decline. A nonlinear encoder can pack information more cleverly than a linear one, but no encoder can beat the counting argument in general: real numbers at finite precision hold a bounded number of bits, while the input's information grows with . The forgetting factor adds a second effect. A recurrent encoder overwrites old information with new, so the first tokens are the worst preserved, exactly the ones a left-to-right decoder needs first. That is why Sutskever et al. reversed the source sentence, and why attention, which skips the bottleneck entirely, made the trick unnecessary.
Worked example: one number for two tokens
Take , , . In one dimension a rotation is , and the figure's encoder gives , so . With noise variance :
- 1. Its eigenvalues are 2 (direction ) and 0 (direction ).
- 2The decoder recovers the sum and nothing of the difference: .
- 3So and : each token is half recovered.
- 4, mean . The figure's library returns the same 0.4994.
- 5Give the decoder both states and instead, and , : the bottleneck is gone.
The output side of the same model is the chain-rule factorisation. If a trained decoder assigns the three reference tokens of a translation the probabilities 0.6, 0.5 and 0.9 given their prefixes, then and the training loss for this pair is nats, or 0.436 per token.
1import torch, torch.nn as nn, torch.nn.functional as F23class EncoderDecoder(nn.Module):4 """Cho et al. (2014): c = last encoder state, fed to every decoder step."""5 def __init__(self, src_vocab, tgt_vocab, emb=256, hid=512):6 super().__init__()7 self.src_emb = nn.Embedding(src_vocab, emb)8 self.tgt_emb = nn.Embedding(tgt_vocab, emb)9 self.encoder = nn.GRU(emb, hid, batch_first=True)10 self.init = nn.Linear(hid, hid)11 self.decoder = nn.GRU(emb + hid, hid, batch_first=True) # input = [y_{t-1}; c]12 self.out = nn.Linear(hid + hid, tgt_vocab) # logits from [s_t; c]1314 def encode(self, src): # src: (N, T) token ids15 _, h_n = self.encoder(self.src_emb(src)) # h_n: (1, N, hid)16 return h_n[-1] # c: (N, hid), the whole input in one vector1718 def forward(self, src, tgt_in): # tgt_in: (N, T') = <sos> y_1 .. y_{T'-1}19 c = self.encode(src)20 s0 = torch.tanh(self.init(c)).unsqueeze(0)21 ctx = c.unsqueeze(1).expand(-1, tgt_in.size(1), -1)22 s, _ = self.decoder(torch.cat([self.tgt_emb(tgt_in), ctx], dim=-1), s0)23 return self.out(torch.cat([s, ctx], dim=-1)) # (N, T', tgt_vocab)2425model = EncoderDecoder(src_vocab=8000, tgt_vocab=8000)26src = torch.randint(0, 8000, (32, 17))27tgt = torch.randint(0, 8000, (32, 12)) # <sos> ... <eos>28logits = model(src, tgt[:, :-1]) # teacher forcing: shifted ground truth29loss = F.cross_entropy(logits.reshape(-1, 8000), tgt[:, 1:].reshape(-1))3031# The Transformer is the same split, but the decoder cross-attends to ALL encoder vectors.32tf = nn.Transformer(d_model=512, nhead=8, num_encoder_layers=6,33 num_decoder_layers=6, batch_first=True)34x, y = torch.randn(32, 17, 512), torch.randn(32, 11, 512) # embedded source / shifted target35memory = tf.encoder(x) # (32, 17, 512): one vector per token36mask = nn.Transformer.generate_square_subsequent_mask(11) # causal mask for the decoder37h = tf.decoder(y, memory, tgt_mask=mask) # (32, 11, 512)Use when
- +Input and output lengths, vocabularies and even modalities are decoupled
- +One principled loss: the conditional log-likelihood , trained end to end
- +The encoder sees the whole input (bidirectionally) before any output is produced
- +Encoder and decoder can be swapped or pre-trained separately (CNN encoder + LSTM decoder, pretrained T5 encoder)
Avoid when
- −A single fixed vector caps what is transmitted: quality falls with input length
- −Autoregressive decoding is sequential: output latency grows with output length
- −Training with ground-truth prefixes but generating from its own outputs causes exposure bias
- −Two networks to size and tune; the decoder may learn to ignore a weak encoder
Check yourself
An RNN encoder–decoder passes a single 256-dimensional vector from encoder to decoder. Translation quality is good for 10-word sentences but poor for 50-word ones. Which change addresses the cause most directly?
Check yourself
In the figure's linear model with d = 4, what is the largest possible mean R² for a sequence of T = 16 tokens?
Remember
- ✓Encoder: input representation . Decoder: a conditional language model .
- ✓Trained end to end with per-token cross-entropy; generation is token by token until an end-of-sequence token.
- ✓The pattern spans modalities: RNN Seq2Seq, Show and Tell captioning, U-Net, autoencoders, Transformer, T5, Whisper.
- ✓A fixed vector of numbers can carry at most tokens' worth of a linear code, so quality falls with input length (Cho et al., 2014).
- ✓Attention and skip connections widen the interface to one vector per input position and remove the bottleneck.