Definition
Sequence-to-sequence (Seq2Seq) learning is the encoder–decoder recipe of Sutskever, Vinyals and Le ("Sequence to Sequence Learning with Neural Networks", NeurIPS 2014): an encoder LSTM reads the input sequence one token at a time, its final state initialises a decoder LSTM, and the decoder emits the output one token at a time, each conditioned on the previous ones, until it produces an end-of-sequence token <eos>.
Before 2014, deep networks needed inputs and outputs of fixed size, and machine translation was the domain of phrase-based statistical systems with many hand-built components. Seq2Seq showed that a single neural network, trained end to end on sentence pairs with nothing but a log-likelihood loss, could translate as well as such a system. The general two-part pattern is the subject of Encoder–decoder architecture; this article is about the Sutskever et al. model and how to train and run it.
It was one of three closely related papers. Kalchbrenner and Blunsom (2013) used a convolutional encoder and a recurrent decoder; Cho et al. (EMNLP 2014) introduced the RNN Encoder–Decoder with GRUs but used it mainly to rescore phrase pairs inside a statistical system. Sutskever et al. made it work as a stand-alone translator by going deep (4-layer LSTMs), big (384M parameters, 8 GPUs, about 10 days of training) and by one simple data trick: reversing the source sentence.
The model
- source tokens, fed in reverse order in the paper ( first)
- final hidden and cell states of all 4 encoder layers: the whole sentence as one fixed-size state
- a start symbol (
<sos>, or in the paper the<EOS>that ends the source) - source and target word embeddings (1000-dimensional)
- output projection to the 80,000-word target vocabulary
Sutskever et al. (2014) setup
- Task
- WMT'14 English→French, a 12M-sentence subset (348M French words, 304M English words)
- Network
- two different LSTMs, 4 layers each, 1000 cells per layer, 1000-dim embeddings
- Vocabularies
- 160,000 source words, 80,000 target words; everything else is
UNK - Parameters
- 384M, of which 64M are recurrent connections (32M encoder, 32M decoder)
- Training
- plain SGD, lr 0.7, halved every half epoch after 5 epochs, 7.5 epochs; batch 128; init ; gradient norm clipped at 5
- Decoding
- left-to-right beam search; beam 2 gave most of the gain over greedy, beam 12 was used for the results
- Best direct result
- 34.81 BLEU, ensemble of 5 reversed-source LSTMs, beam 12 (Table 1)
Training loss and the two special tokens
Training maximises the log-probability of the correct translation given the source over the training set : . Per pair this is a sum of token cross-entropies, computed with the ground-truth prefix fed to the decoder (Teacher forcing):
- fed as the first decoder input, so the first word has something to be conditioned on
- appended to every target and predicted like any other token; it is how the model chooses the output length
- the number of predicted tokens: the words plus
<eos>
In code the decoder input is the target shifted right, [<sos>, y₁, …, y_T'], and the label is [y₁, …, y_T', <eos>]; padding positions are excluded with ignore_index. Because the model must emit <eos>, the distribution over output lengths is learned, not fixed. At test time the decoder runs on its own predictions:
Greedy inference loop
- 1Encode the (reversed) source; set the decoder state to the encoder's final and the input token to
<sos>. - 2Run one decoder step and compute over the vocabulary.
- 3Pick (greedy) and append it to the output.
- 4If
<eos>or a length limit is reached, stop; otherwise feed back as the next input and go to step 2.
Greedy decoding commits to the single best token at every step and can paint itself into a corner. Sutskever et al. kept the best partial hypotheses instead (see Beam search). Beam 1 versus 12 cost the ensemble 1.8 BLEU (33.00 vs 34.81).
Why reversing the source helps
For English→French the alignment is roughly monotone: the first target word mostly depends on the first source words. Read in order, the first source word is seen steps before the decoder starts, so every source word is the same long distance from the target word it aligns with. Read reversed, the first source word is the last thing the encoder sees, right next to the first decoder step. The average distance does not change, but the shortest ones become very short, and Sutskever et al. argue that these short dependencies make it much easier for backpropagation to "establish communication" between source and target early in training. Reversing cut the test perplexity of a single LSTM from 5.8 to 4.7 and lifted its decoded BLEU from 25.9 to 30.6.
| word i | 1 | 2 | 3 | 4 | 5 | mean |
|---|---|---|---|---|---|---|
| source read in order | 5 | 5 | 5 | 5 | 5 | 5 |
| source reversed | 1 | 3 | 5 | 7 | 9 | 5 |
The toy exaggerates the effect, because a single small tanh RNN copying in order must keep a first-in-first-out queue in its state, while the reversed model only has to unwind a stack, most recent token first. Real translation is not perfectly monotone and real LSTMs are far stronger, which is why the paper's gain was 4.7 BLEU, not the difference between working and not working. Attention, a year later, made the trick unnecessary: the decoder can read any source position directly, whatever the order.
Results
| system | BLEU |
|---|---|
| Phrase-based SMT baseline (Schwenk, 2014) | 33.30 |
| Single forward LSTM, beam 12 | 26.17 |
| Single reversed LSTM, beam 12 | 30.59 |
| Ensemble of 5 reversed LSTMs, beam 1 | 33.00 |
| Ensemble of 5 reversed LSTMs, beam 12 | 34.81 |
| Baseline + rescoring its 1000-best lists with the 5-LSTM ensemble (Table 2) | 36.5 |
| Best WMT'14 result (Table 2) | 37.0 |
The direct translation beat the phrase-based baseline by 1.5 BLEU even though the LSTM had to emit UNK for every out-of-vocabulary word and was penalised for it. The paper also found, against expectations, that the LSTM did not degrade badly on long sentences, and a PCA of the encoder states showed that sentences with the same meaning in active and passive voice landed close together while word-order changes that alter meaning moved them apart.
Worked example: greedy decoding in the figure
After 4,000 training steps the figure's default sentence is F A E E C C B. The reversed model reads B C C E E A F and decodes:
| step j | decoder input | arg-max output | p(output) |
|---|---|---|---|
| 1 | <sos> | f | 0.997 |
| 2 | f | a | 0.928 |
| 3 | a | e | 0.991 |
| 4 | e | e | 0.999 |
| 5 | e | c | 0.999 |
| 6 | c | c | 0.977 |
| 7 | c | b | 0.996 |
| 8 | b | <eos> | 1.000 |
Every step is correct, and the teacher-forced loss of this pair would be about nats. The in-order model also gets step 1 right (f, 0.956) but at step 3 it outputs a again with probability 0.218, and from then on it is conditioning on its own mistake: the output is f a a f f d f <eos>. It does get the length right; the <eos> decision is easy because the length is held in the state.
1import torch, torch.nn as nn, torch.nn.functional as F23PAD, SOS, EOS = 0, 1, 245class Seq2Seq(nn.Module):6 def __init__(self, src_vocab, tgt_vocab, emb=1000, hid=1000, layers=4):7 super().__init__()8 self.src_emb = nn.Embedding(src_vocab, emb, padding_idx=PAD)9 self.tgt_emb = nn.Embedding(tgt_vocab, emb, padding_idx=PAD)10 self.encoder = nn.LSTM(emb, hid, num_layers=layers, batch_first=True)11 self.decoder = nn.LSTM(emb, hid, num_layers=layers, batch_first=True)12 self.out = nn.Linear(hid, tgt_vocab)13 for p in self.parameters(): # the paper's init14 nn.init.uniform_(p, -0.08, 0.08)1516 def encode(self, src, src_len):17 packed = nn.utils.rnn.pack_padded_sequence(18 self.src_emb(src), src_len.cpu(), batch_first=True, enforce_sorted=False)19 _, (h, c) = self.encoder(packed) # each (layers, N, hid)20 return h, c # true last state of every sequence2122 def forward(self, src, src_len, tgt_in): # teacher forcing23 state = self.encode(src, src_len)24 dec, _ = self.decoder(self.tgt_emb(tgt_in), state)25 return self.out(dec) # (N, T', vocab)2627 @torch.no_grad()28 def greedy(self, src, src_len, max_len=100):29 state = self.encode(src, src_len)30 y = torch.full((src.size(0), 1), SOS, dtype=torch.long, device=src.device)31 outs, done = [], torch.zeros(src.size(0), dtype=torch.bool, device=src.device)32 for _ in range(max_len):33 dec, state = self.decoder(self.tgt_emb(y), state) # one step, carry (h, c)34 y = self.out(dec[:, -1]).argmax(-1, keepdim=True) # feed back own prediction35 outs.append(y)36 done |= y.squeeze(1) == EOS37 if done.all():38 break39 return torch.cat(outs, dim=1)4041def reverse_source(tokens): # applied per sentence, before padding42 return tokens[::-1]4344model = Seq2Seq(src_vocab=160_000, tgt_vocab=80_000)45opt = torch.optim.SGD(model.parameters(), lr=0.7)4647def train_step(src, src_len, tgt): # tgt = <sos> y_1 .. y_T' <eos>, padded48 logits = model(src, src_len, tgt[:, :-1]) # inputs: <sos> y_1 .. y_T'49 loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)),50 tgt[:, 1:].reshape(-1), # labels: y_1 .. y_T' <eos>51 ignore_index=PAD)52 opt.zero_grad()53 loss.backward()54 torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)55 opt.step()56 return loss.item()Use when
- +End to end: one network and one log-likelihood loss replace a pipeline of hand-built translation components
- +Handles any input and output lengths; the model chooses the output length by emitting <eos>
- +Reversing the source is a free, one-line change that markedly eases optimisation
- +The same recipe works for parsing, captioning, summarisation and speech
Avoid when
- −Everything passes through one fixed-size state: a bottleneck for long or detailed inputs
- −Sequential in both encoder and decoder: slow to train and to decode
- −Large closed vocabularies (80k–160k words) with UNK for everything else
- −Teacher-forced training versus free-running inference causes exposure bias
Check yourself
A target sentence has 6 words. How many decoder steps contribute a cross-entropy term to its teacher-forced loss, and what is the last label?
Check yourself
Reversing the source sentence leaves the average distance between aligned source and target words unchanged. Why did it still help Sutskever et al.?
Remember
- ✓Seq2Seq (Sutskever, Vinyals & Le 2014): encoder LSTM → final state → decoder LSTM, trained end to end on sentence pairs.
- ✓Loss: with decoder input
<sos>+ target and labels target +<eos>. - ✓Inference feeds back the decoder's own predictions until <eos>, greedily or with beam search.
- ✓Reversing the source gives short source–target lags and lifted a single LSTM from 25.9 to 30.6 BLEU.
- ✓4-layer, 1000-unit LSTMs, 384M parameters; a 5-model ensemble with beam 12 scored 34.81 BLEU on WMT'14 En→Fr vs 33.30 for phrase-based SMT.