# Sequence-to-sequence (Seq2Seq)

> Sutskever, Vinyals and Le (2014): one deep LSTM reads the source sentence, reversed, into its final state; a second LSTM generates the translation from it until <eos>. With 4 layers and a 5-model ensemble it reached 34.8 BLEU on WMT'14 English→French.

Source: https://www.mlfoundry.in/learn/sequence-to-sequence-seq2seq  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 13 min read · updated 2026-10-12

**Sequence-to-sequence (Seq2Seq):** learning is the encoder–decoder recipe of Sutskever, Vinyals and Le ("Sequence to Sequence Learning with Neural Networks", NeurIPS 2014): an **encoder LSTM** reads the input sequence $x_1..x_T$ one token at a time, its final state $(h_T,c_T)$ initialises a **decoder LSTM**, and the decoder emits the output $y_1..y_{T'}$ one token at a time, each conditioned on the previous ones, until it produces an end-of-sequence token `<eos>`.

Before 2014, deep networks needed inputs and outputs of fixed size, and machine translation was the domain of phrase-based statistical systems with many hand-built components. Seq2Seq showed that a single neural network, trained end to end on sentence pairs with nothing but a log-likelihood loss, could translate as well as such a system. The general two-part pattern is the subject of **Encoder–decoder architecture**; this article is about the Sutskever et al. model and how to train and run it.

It was one of three closely related papers. Kalchbrenner and Blunsom (2013) used a convolutional encoder and a recurrent decoder; Cho et al. (EMNLP 2014) introduced the RNN Encoder–Decoder with GRUs but used it mainly to rescore phrase pairs inside a statistical system. Sutskever et al. made it work as a stand-alone translator by going **deep** (4-layer LSTMs), **big** (384M parameters, 8 GPUs, about 10 days of training) and by one simple data trick: **reversing the source sentence**.

## The model

$$
\begin{aligned}(h_t,c_t)&=\mathrm{LSTM}_{\text{enc}}\big(e(x_t),\,(h_{t-1},c_{t-1})\big),\qquad (s_0,m_0)=(h_T,c_T)\\ (s_j,m_j)&=\mathrm{LSTM}_{\text{dec}}\big(e'(y_{j-1}),\,(s_{j-1},m_{j-1})\big),\qquad p(y_j\mid y_{<j},x)=\mathrm{softmax}(W_o s_j)\end{aligned}
$$

- $x_1..x_T$: source tokens, fed in **reverse** order in the paper ($x_T$ first)
- $(h_T,c_T)$: final hidden and cell states of all 4 encoder layers: the whole sentence as one fixed-size state
- $y_0$: a start symbol (`<sos>`, or in the paper the `<EOS>` that ends the source)
- $e,\ e'$: source and target word embeddings (1000-dimensional)
- $W_o$: output projection to the 80,000-word target vocabulary

**Sutskever et al. (2014) setup**

- **Task:** WMT'14 English→French, a 12M-sentence subset (348M French words, 304M English words)
- **Network:** two different LSTMs, 4 layers each, 1000 cells per layer, 1000-dim embeddings
- **Vocabularies:** 160,000 source words, 80,000 target words; everything else is `UNK`
- **Parameters:** 384M, of which 64M are recurrent connections (32M encoder, 32M decoder)
- **Training:** plain SGD, lr 0.7, halved every half epoch after 5 epochs, 7.5 epochs; batch 128; init $U(-0.08,0.08)$; gradient norm clipped at 5
- **Decoding:** left-to-right beam search; beam 2 gave most of the gain over greedy, beam 12 was used for the results
- **Best direct result:** **34.81 BLEU**, ensemble of 5 reversed-source LSTMs, beam 12 (Table 1)

## Training loss and the two special tokens

Training maximises the log-probability of the correct translation $T$ given the source $S$ over the training set $\mathcal S$: $\frac1{|\mathcal S|}\sum_{(T,S)\in\mathcal S}\log p(T\mid S)$. Per pair this is a sum of token cross-entropies, computed with the ground-truth prefix fed to the decoder (**Teacher forcing**):

$$
\mathcal L(x,y)=-\sum_{j=1}^{T'+1}\log p\big(y_j\mid y_0,\dots,y_{j-1},\,x\big),\qquad y_0=\texttt{<sos>},\ \ y_{T'+1}=\texttt{<eos>}
$$

- $\texttt{<sos>}$: fed as the first decoder input, so the first word has something to be conditioned on
- $\texttt{<eos>}$: appended to every target and predicted like any other token; it is how the model chooses the output length
- $T'+1$: the number of predicted tokens: the $T'$ words plus `<eos>`

In code the decoder input is the target shifted right, `[<sos>, y₁, …, y_T']`, and the label is `[y₁, …, y_T', <eos>]`; padding positions are excluded with `ignore_index`. Because the model must emit `<eos>`, the distribution over output lengths is learned, not fixed. At test time the decoder runs on its own predictions:

**Greedy inference loop**

1. Encode the (reversed) source; set the decoder state to the encoder's final $(h,c)$ and the input token to `<sos>`.
2. Run one decoder step and compute $p(y_j\mid y_{<j},x)$ over the vocabulary.
3. Pick $\hat y_j=\arg\max$ (greedy) and append it to the output.
4. If $\hat y_j=$ `<eos>` or a length limit is reached, stop; otherwise feed $\hat y_j$ back as the next input and go to step 2.

Greedy decoding commits to the single best token at every step and can paint itself into a corner. Sutskever et al. kept the $B$ best partial hypotheses instead (see **Beam search**). Beam 1 versus 12 cost the ensemble 1.8 BLEU (33.00 vs 34.81).

## Why reversing the source helps

For English→French the alignment is roughly monotone: the first target word mostly depends on the first source words. Read in order, the first source word is seen $T$ steps before the decoder starts, so every source word is the same long distance from the target word it aligns with. Read reversed, the first source word is the **last** thing the encoder sees, right next to the first decoder step. The average distance does not change, but the shortest ones become very short, and Sutskever et al. argue that these short dependencies make it much easier for backpropagation to "establish communication" between source and target early in training. Reversing cut the test perplexity of a single LSTM from 5.8 to 4.7 and lifted its decoded BLEU from 25.9 to 30.6.

Distance, in RNN steps, between reading source word i and emitting target word i for a 5-word sentence (the figure computes it for its current sentence)

| word i | 1 | 2 | 3 | 4 | 5 | mean |
| --- | --- | --- | --- | --- | --- | --- |
| source read in order | 5 | 5 | 5 | 5 | 5 | 5 |
| source reversed | 1 | 3 | 5 | 7 | 9 | 5 |

> Figure: Two tiny encoder–decoders (single-layer tanh RNNs, 32 units, 2,759 parameters each) trained live on the same batches of a toy translation task: a sentence of 3 to 8 capital letters must be rewritten in lower case, a monotone alignment like English→French. One reads the source in order, the other reversed. Press train both: after 1,000 Adam steps the reversed model translates 75.4% of 240 held-out sentences exactly with greedy decoding, the in-order model 0.0%; after 4,000 steps it is 93.8% vs 4.6%, and the per-token loss is 0.059 vs 1.022 nats. The lower panel steps through greedy decoding of one sentence: the decoder starts from <sos>, feeds back each arg-max token, and stops at <eos>.

The toy exaggerates the effect, because a single small tanh RNN copying in order must keep a first-in-first-out queue in its state, while the reversed model only has to unwind a stack, most recent token first. Real translation is not perfectly monotone and real LSTMs are far stronger, which is why the paper's gain was 4.7 BLEU, not the difference between working and not working. Attention, a year later, made the trick unnecessary: the decoder can read any source position directly, whatever the order.

## Results

WMT'14 English→French test BLEU, Sutskever et al. (2014), Tables 1 and 2

| system | BLEU |
| --- | --- |
| Phrase-based SMT baseline (Schwenk, 2014) | 33.30 |
| Single forward LSTM, beam 12 | 26.17 |
| Single reversed LSTM, beam 12 | 30.59 |
| Ensemble of 5 reversed LSTMs, beam 1 | 33.00 |
| Ensemble of 5 reversed LSTMs, beam 12 | 34.81 |
| Baseline + rescoring its 1000-best lists with the 5-LSTM ensemble (Table 2) | 36.5 |
| Best WMT'14 result (Table 2) | 37.0 |

The direct translation beat the phrase-based baseline by 1.5 BLEU even though the LSTM had to emit `UNK` for every out-of-vocabulary word and was penalised for it. The paper also found, against expectations, that the LSTM did **not** degrade badly on long sentences, and a PCA of the encoder states showed that sentences with the same meaning in active and passive voice landed close together while word-order changes that alter meaning moved them apart.

## Worked example: greedy decoding in the figure

After 4,000 training steps the figure's default sentence is `F A E E C C B`. The reversed model reads `B C C E E A F` and decodes:

Greedy decoding with the reversed-source model after 4,000 steps (values from the figure's library)

| step j | decoder input | arg-max output | p(output) |
| --- | --- | --- | --- |
| 1 | <sos> | f | 0.997 |
| 2 | f | a | 0.928 |
| 3 | a | e | 0.991 |
| 4 | e | e | 0.999 |
| 5 | e | c | 0.999 |
| 6 | c | c | 0.977 |
| 7 | c | b | 0.996 |
| 8 | b | <eos> | 1.000 |

Every step is correct, and the teacher-forced loss of this pair would be about $-\sum_j\ln p_j\approx0.003+0.075+0.009+0.001+0.001+0.023+0.004+0.000=0.116$ nats. The in-order model also gets step 1 right (`f`, 0.956) but at step 3 it outputs `a` again with probability 0.218, and from then on it is conditioning on its own mistake: the output is `f a a f f d f <eos>`. It does get the length right; the `<eos>` decision is easy because the length is held in the state.

```python
import torch, torch.nn as nn, torch.nn.functional as F

PAD, SOS, EOS = 0, 1, 2

class Seq2Seq(nn.Module):
    def __init__(self, src_vocab, tgt_vocab, emb=1000, hid=1000, layers=4):
        super().__init__()
        self.src_emb = nn.Embedding(src_vocab, emb, padding_idx=PAD)
        self.tgt_emb = nn.Embedding(tgt_vocab, emb, padding_idx=PAD)
        self.encoder = nn.LSTM(emb, hid, num_layers=layers, batch_first=True)
        self.decoder = nn.LSTM(emb, hid, num_layers=layers, batch_first=True)
        self.out = nn.Linear(hid, tgt_vocab)
        for p in self.parameters():                       # the paper's init
            nn.init.uniform_(p, -0.08, 0.08)

    def encode(self, src, src_len):
        packed = nn.utils.rnn.pack_padded_sequence(
            self.src_emb(src), src_len.cpu(), batch_first=True, enforce_sorted=False)
        _, (h, c) = self.encoder(packed)                  # each (layers, N, hid)
        return h, c                                       # true last state of every sequence

    def forward(self, src, src_len, tgt_in):              # teacher forcing
        state = self.encode(src, src_len)
        dec, _ = self.decoder(self.tgt_emb(tgt_in), state)
        return self.out(dec)                              # (N, T', vocab)

    @torch.no_grad()
    def greedy(self, src, src_len, max_len=100):
        state = self.encode(src, src_len)
        y = torch.full((src.size(0), 1), SOS, dtype=torch.long, device=src.device)
        outs, done = [], torch.zeros(src.size(0), dtype=torch.bool, device=src.device)
        for _ in range(max_len):
            dec, state = self.decoder(self.tgt_emb(y), state)   # one step, carry (h, c)
            y = self.out(dec[:, -1]).argmax(-1, keepdim=True)   # feed back own prediction
            outs.append(y)
            done |= y.squeeze(1) == EOS
            if done.all():
                break
        return torch.cat(outs, dim=1)

def reverse_source(tokens):                               # applied per sentence, before padding
    return tokens[::-1]

model = Seq2Seq(src_vocab=160_000, tgt_vocab=80_000)
opt = torch.optim.SGD(model.parameters(), lr=0.7)

def train_step(src, src_len, tgt):                        # tgt = <sos> y_1 .. y_T' <eos>, padded
    logits = model(src, src_len, tgt[:, :-1])             # inputs:  <sos> y_1 .. y_T'
    loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)),
                           tgt[:, 1:].reshape(-1),        # labels:  y_1 .. y_T' <eos>
                           ignore_index=PAD)
    opt.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
    opt.step()
    return loss.item()
```

Seq2Seq in PyTorch: reversed source, 4-layer LSTMs, teacher-forced training, greedy decoding

> **Reverse before padding, and use packed sequences:** Flipping a right-padded batch with `torch.flip` puts the padding **first**, so the encoder reads a run of `<pad>` tokens before the sentence, and its final state differs from the unpadded one. Reverse each sentence before padding, and pack the batch with `pack_padded_sequence` so that `h_n` is the state at each sentence's true last token. Also mask `<pad>` in the loss (`ignore_index`), or the model learns to predict padding.

> **Training and inference see different inputs:** Training feeds the decoder the reference prefix; greedy or beam decoding feeds it the model's own output. A single early mistake therefore puts the decoder in states it never saw in training, as the in-order model in the worked example shows. This is the exposure-bias problem discussed in **Teacher forcing** and **Scheduled sampling**.

> **What came next:** Bahdanau, Cho and Bengio's attention (2015) let the decoder look at every encoder state instead of one final vector, and Google's GNMT (Wu et al., 2016) combined 8-layer LSTMs with attention and residual connections for production translation. The Transformer (2017) kept the Seq2Seq training recipe (encoder, autoregressive decoder, teacher-forced cross-entropy, beam search) and replaced the recurrence with attention.

**Strengths**

- End to end: one network and one log-likelihood loss replace a pipeline of hand-built translation components
- Handles any input and output lengths; the model chooses the output length by emitting <eos>
- Reversing the source is a free, one-line change that markedly eases optimisation
- The same recipe works for parsing, captioning, summarisation and speech

**Limitations**

- Everything passes through one fixed-size state: a bottleneck for long or detailed inputs
- Sequential in both encoder and decoder: slow to train and to decode
- Large closed vocabularies (80k–160k words) with UNK for everything else
- Teacher-forced training versus free-running inference causes exposure bias

**Question:** A target sentence has 6 words. How many decoder steps contribute a cross-entropy term to its teacher-forced loss, and what is the last label?

- A. 6, the last word
- B. 7, <eos>
- C. 7, <sos>
- D. 5, the last word

**Answer:** B. 7, <eos> — The decoder inputs are `<sos>` plus the 6 words (7 steps) and the labels are the 6 words plus `<eos>`. Predicting `<eos>` is what teaches the model when to stop; `<sos>` is only ever an input.

**Question:** Reversing the source sentence leaves the average distance between aligned source and target words unchanged. Why did it still help Sutskever et al.?

- A. It created many short dependencies between the first source and first target words, easing early optimisation
- B. It shortened every sentence
- C. It made the LSTM bidirectional
- D. French is written right to left

**Answer:** A. It created many short dependencies between the first source and first target words, easing early optimisation — For a 5-word sentence the lags go from 5, 5, 5, 5, 5 to 1, 3, 5, 7, 9. The mean is still 5, but the first words are now adjacent across the encoder–decoder boundary, so gradients connect them easily and the model gets the start of the translation right, which then conditions everything after it.

**Key points**

- Seq2Seq (Sutskever, Vinyals & Le 2014): encoder LSTM → final state → decoder LSTM, trained end to end on sentence pairs.
- Loss: $-\sum_j\log p(y_j\mid y_{<j},x)$ with decoder input `<sos>` + target and labels target + `<eos>`.
- Inference feeds back the decoder's own predictions until <eos>, greedily or with beam search.
- Reversing the source gives short source–target lags and lifted a single LSTM from 25.9 to 30.6 BLEU.
- 4-layer, 1000-unit LSTMs, 384M parameters; a 5-model ensemble with beam 12 scored 34.81 BLEU on WMT'14 En→Fr vs 33.30 for phrase-based SMT.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
