Sequence-to-sequence (Seq2Seq)

Sutskever, Vinyals and Le (2014): one deep LSTM reads the source sentence, reversed, into its final state; a second LSTM generates the translation from it until <eos>. With 4 layers and a 5-model ensemble it reached 34.8 BLEU on WMT'14 English→French.

13 min read1090 wordscore0 viewsupdated 2026-10-12

Definition

Sequence-to-sequence (Seq2Seq) learning is the encoder–decoder recipe of Sutskever, Vinyals and Le ("Sequence to Sequence Learning with Neural Networks", NeurIPS 2014): an encoder LSTM reads the input sequence x1..xTx_1..x_T one token at a time, its final state (hT,cT)(h_T,c_T) initialises a decoder LSTM, and the decoder emits the output y1..yT′y_1..y_{T'} one token at a time, each conditioned on the previous ones, until it produces an end-of-sequence token <eos>.

Before 2014, deep networks needed inputs and outputs of fixed size, and machine translation was the domain of phrase-based statistical systems with many hand-built components. Seq2Seq showed that a single neural network, trained end to end on sentence pairs with nothing but a log-likelihood loss, could translate as well as such a system. The general two-part pattern is the subject of Encoder–decoder architecture; this article is about the Sutskever et al. model and how to train and run it.

It was one of three closely related papers. Kalchbrenner and Blunsom (2013) used a convolutional encoder and a recurrent decoder; Cho et al. (EMNLP 2014) introduced the RNN Encoder–Decoder with GRUs but used it mainly to rescore phrase pairs inside a statistical system. Sutskever et al. made it work as a stand-alone translator by going deep (4-layer LSTMs), big (384M parameters, 8 GPUs, about 10 days of training) and by one simple data trick: reversing the source sentence.

The model

(ht,ct)=LSTMenc(e(xt), (ht−1,ct−1)),(s0,m0)=(hT,cT)(sj,mj)=LSTMdec(e′(yj−1), (sj−1,mj−1)),p(yj∣y<j,x)=softmax(Wosj)\begin{aligned}(h_t,c_t)&=\mathrm{LSTM}_{\text{enc}}\big(e(x_t),\,(h_{t-1},c_{t-1})\big),\qquad (s_0,m_0)=(h_T,c_T)\\ (s_j,m_j)&=\mathrm{LSTM}_{\text{dec}}\big(e'(y_{j-1}),\,(s_{j-1},m_{j-1})\big),\qquad p(y_j\mid y_{<j},x)=\mathrm{softmax}(W_o s_j)\end{aligned}
x1..xTx_1..x_T
source tokens, fed in reverse order in the paper (xTx_T first)
(hT,cT)(h_T,c_T)
final hidden and cell states of all 4 encoder layers: the whole sentence as one fixed-size state
y0y_0
a start symbol (<sos>, or in the paper the <EOS> that ends the source)
e, e′e,\ e'
source and target word embeddings (1000-dimensional)
WoW_o
output projection to the 80,000-word target vocabulary

Sutskever et al. (2014) setup

Task
WMT'14 English→French, a 12M-sentence subset (348M French words, 304M English words)
Network
two different LSTMs, 4 layers each, 1000 cells per layer, 1000-dim embeddings
Vocabularies
160,000 source words, 80,000 target words; everything else is UNK
Parameters
384M, of which 64M are recurrent connections (32M encoder, 32M decoder)
Training
plain SGD, lr 0.7, halved every half epoch after 5 epochs, 7.5 epochs; batch 128; init U(−0.08,0.08)U(-0.08,0.08); gradient norm clipped at 5
Decoding
left-to-right beam search; beam 2 gave most of the gain over greedy, beam 12 was used for the results
Best direct result
34.81 BLEU, ensemble of 5 reversed-source LSTMs, beam 12 (Table 1)

Training loss and the two special tokens

Training maximises the log-probability of the correct translation TT given the source SS over the training set S\mathcal S: 1∣S∣∑(T,S)∈Slog⁡p(T∣S)\frac1{|\mathcal S|}\sum_{(T,S)\in\mathcal S}\log p(T\mid S). Per pair this is a sum of token cross-entropies, computed with the ground-truth prefix fed to the decoder (Teacher forcing):

L(x,y)=−∑j=1T′+1log⁡p(yj∣y0,…,yj−1, x),y0=<sos>,  yT′+1=<eos>\mathcal L(x,y)=-\sum_{j=1}^{T'+1}\log p\big(y_j\mid y_0,\dots,y_{j-1},\,x\big),\qquad y_0=\texttt{<sos>},\ \ y_{T'+1}=\texttt{<eos>}
<sos>\texttt{<sos>}
fed as the first decoder input, so the first word has something to be conditioned on
<eos>\texttt{<eos>}
appended to every target and predicted like any other token; it is how the model chooses the output length
T′+1T'+1
the number of predicted tokens: the T′T' words plus <eos>

In code the decoder input is the target shifted right, [<sos>, y₁, …, y_T'], and the label is [y₁, …, y_T', <eos>]; padding positions are excluded with ignore_index. Because the model must emit <eos>, the distribution over output lengths is learned, not fixed. At test time the decoder runs on its own predictions:

Greedy inference loop

  1. 1Encode the (reversed) source; set the decoder state to the encoder's final (h,c)(h,c) and the input token to <sos>.
  2. 2Run one decoder step and compute p(yj∣y<j,x)p(y_j\mid y_{<j},x) over the vocabulary.
  3. 3Pick y^j=arg⁡max⁡\hat y_j=\arg\max (greedy) and append it to the output.
  4. 4If y^j=\hat y_j= <eos> or a length limit is reached, stop; otherwise feed y^j\hat y_j back as the next input and go to step 2.

Greedy decoding commits to the single best token at every step and can paint itself into a corner. Sutskever et al. kept the BB best partial hypotheses instead (see Beam search). Beam 1 versus 12 cost the ensemble 1.8 BLEU (33.00 vs 34.81).

Why reversing the source helps

For English→French the alignment is roughly monotone: the first target word mostly depends on the first source words. Read in order, the first source word is seen TT steps before the decoder starts, so every source word is the same long distance from the target word it aligns with. Read reversed, the first source word is the last thing the encoder sees, right next to the first decoder step. The average distance does not change, but the shortest ones become very short, and Sutskever et al. argue that these short dependencies make it much easier for backpropagation to "establish communication" between source and target early in training. Reversing cut the test perplexity of a single LSTM from 5.8 to 4.7 and lifted its decoded BLEU from 25.9 to 30.6.

word i12345mean
source read in order555555
source reversed135795
Distance, in RNN steps, between reading source word i and emitting target word i for a 5-word sentence (the figure computes it for its current sentence)
teacher-forced loss (nats/token) · step 010.10.014000 stepsin order –reversed –greedy exact match, held-out (step 0)L=3L=4L=5L=6L=7L=8all lengths: 0.0% vs 0.0%encoder reads (reversed)greedy decoder: feeds back its own outputBCCEEAF→sosaa?e?reference: f a e e c c b eoslag from source word i to target word i (RNN steps): 1, 3, 5, 7, 9, 11, 13 · mean 7.0step 1: fed <sos>, output a<eos>0.16a0.17b0.11c0.13d0.15e0.16f0.12translation wrongbars: p(yⱼ | y<ⱼ, x), green = reference━ in order ━ reversed source · tanh RNN, H = 32, 6 letters, lengths 3–8, same batches
decode with
Adam, lr 0.005, clip 1, batch 16, eval every 100
fig.Two tiny encoder–decoders (single-layer tanh RNNs, 32 units, 2,759 parameters each) trained live on the same batches of a toy translation task: a sentence of 3 to 8 capital letters must be rewritten in lower case, a monotone alignment like English→French. One reads the source in order, the other reversed. Press train both: after 1,000 Adam steps the reversed model translates 75.4% of 240 held-out sentences exactly with greedy decoding, the in-order model 0.0%; after 4,000 steps it is 93.8% vs 4.6%, and the per-token loss is 0.059 vs 1.022 nats. The lower panel steps through greedy decoding of one sentence: the decoder starts from <sos>, feeds back each arg-max token, and stops at <eos>.

The toy exaggerates the effect, because a single small tanh RNN copying in order must keep a first-in-first-out queue in its state, while the reversed model only has to unwind a stack, most recent token first. Real translation is not perfectly monotone and real LSTMs are far stronger, which is why the paper's gain was 4.7 BLEU, not the difference between working and not working. Attention, a year later, made the trick unnecessary: the decoder can read any source position directly, whatever the order.

Results

systemBLEU
Phrase-based SMT baseline (Schwenk, 2014)33.30
Single forward LSTM, beam 1226.17
Single reversed LSTM, beam 1230.59
Ensemble of 5 reversed LSTMs, beam 133.00
Ensemble of 5 reversed LSTMs, beam 1234.81
Baseline + rescoring its 1000-best lists with the 5-LSTM ensemble (Table 2)36.5
Best WMT'14 result (Table 2)37.0
WMT'14 English→French test BLEU, Sutskever et al. (2014), Tables 1 and 2

The direct translation beat the phrase-based baseline by 1.5 BLEU even though the LSTM had to emit UNK for every out-of-vocabulary word and was penalised for it. The paper also found, against expectations, that the LSTM did not degrade badly on long sentences, and a PCA of the encoder states showed that sentences with the same meaning in active and passive voice landed close together while word-order changes that alter meaning moved them apart.

Worked example: greedy decoding in the figure

After 4,000 training steps the figure's default sentence is F A E E C C B. The reversed model reads B C C E E A F and decodes:

step jdecoder inputarg-max outputp(output)
1<sos>f0.997
2fa0.928
3ae0.991
4ee0.999
5ec0.999
6cc0.977
7cb0.996
8b<eos>1.000
Greedy decoding with the reversed-source model after 4,000 steps (values from the figure's library)

Every step is correct, and the teacher-forced loss of this pair would be about −∑jln⁡pj≈0.003+0.075+0.009+0.001+0.001+0.023+0.004+0.000=0.116-\sum_j\ln p_j\approx0.003+0.075+0.009+0.001+0.001+0.023+0.004+0.000=0.116 nats. The in-order model also gets step 1 right (f, 0.956) but at step 3 it outputs a again with probability 0.218, and from then on it is conditioning on its own mistake: the output is f a a f f d f <eos>. It does get the length right; the <eos> decision is easy because the length is held in the state.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3PAD, SOS, EOS = 0, 1, 2
4
5class Seq2Seq(nn.Module):
6 def __init__(self, src_vocab, tgt_vocab, emb=1000, hid=1000, layers=4):
7 super().__init__()
8 self.src_emb = nn.Embedding(src_vocab, emb, padding_idx=PAD)
9 self.tgt_emb = nn.Embedding(tgt_vocab, emb, padding_idx=PAD)
10 self.encoder = nn.LSTM(emb, hid, num_layers=layers, batch_first=True)
11 self.decoder = nn.LSTM(emb, hid, num_layers=layers, batch_first=True)
12 self.out = nn.Linear(hid, tgt_vocab)
13 for p in self.parameters(): # the paper's init
14 nn.init.uniform_(p, -0.08, 0.08)
15
16 def encode(self, src, src_len):
17 packed = nn.utils.rnn.pack_padded_sequence(
18 self.src_emb(src), src_len.cpu(), batch_first=True, enforce_sorted=False)
19 _, (h, c) = self.encoder(packed) # each (layers, N, hid)
20 return h, c # true last state of every sequence
21
22 def forward(self, src, src_len, tgt_in): # teacher forcing
23 state = self.encode(src, src_len)
24 dec, _ = self.decoder(self.tgt_emb(tgt_in), state)
25 return self.out(dec) # (N, T', vocab)
26
27 @torch.no_grad()
28 def greedy(self, src, src_len, max_len=100):
29 state = self.encode(src, src_len)
30 y = torch.full((src.size(0), 1), SOS, dtype=torch.long, device=src.device)
31 outs, done = [], torch.zeros(src.size(0), dtype=torch.bool, device=src.device)
32 for _ in range(max_len):
33 dec, state = self.decoder(self.tgt_emb(y), state) # one step, carry (h, c)
34 y = self.out(dec[:, -1]).argmax(-1, keepdim=True) # feed back own prediction
35 outs.append(y)
36 done |= y.squeeze(1) == EOS
37 if done.all():
38 break
39 return torch.cat(outs, dim=1)
40
41def reverse_source(tokens): # applied per sentence, before padding
42 return tokens[::-1]
43
44model = Seq2Seq(src_vocab=160_000, tgt_vocab=80_000)
45opt = torch.optim.SGD(model.parameters(), lr=0.7)
46
47def train_step(src, src_len, tgt): # tgt = <sos> y_1 .. y_T' <eos>, padded
48 logits = model(src, src_len, tgt[:, :-1]) # inputs: <sos> y_1 .. y_T'
49 loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)),
50 tgt[:, 1:].reshape(-1), # labels: y_1 .. y_T' <eos>
51 ignore_index=PAD)
52 opt.zero_grad()
53 loss.backward()
54 torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
55 opt.step()
56 return loss.item()
Seq2Seq in PyTorch: reversed source, 4-layer LSTMs, teacher-forced training, greedy decoding

Use when

  • +End to end: one network and one log-likelihood loss replace a pipeline of hand-built translation components
  • +Handles any input and output lengths; the model chooses the output length by emitting <eos>
  • +Reversing the source is a free, one-line change that markedly eases optimisation
  • +The same recipe works for parsing, captioning, summarisation and speech

Avoid when

  • −Everything passes through one fixed-size state: a bottleneck for long or detailed inputs
  • −Sequential in both encoder and decoder: slow to train and to decode
  • −Large closed vocabularies (80k–160k words) with UNK for everything else
  • −Teacher-forced training versus free-running inference causes exposure bias

Check yourself

A target sentence has 6 words. How many decoder steps contribute a cross-entropy term to its teacher-forced loss, and what is the last label?

Check yourself

Reversing the source sentence leaves the average distance between aligned source and target words unchanged. Why did it still help Sutskever et al.?

Remember

  • ✓Seq2Seq (Sutskever, Vinyals & Le 2014): encoder LSTM → final state → decoder LSTM, trained end to end on sentence pairs.
  • ✓Loss: −∑jlog⁡p(yj∣y<j,x)-\sum_j\log p(y_j\mid y_{<j},x) with decoder input <sos> + target and labels target + <eos>.
  • ✓Inference feeds back the decoder's own predictions until <eos>, greedily or with beam search.
  • ✓Reversing the source gives short source–target lags and lifted a single LSTM from 25.9 to 30.6 BLEU.
  • ✓4-layer, 1000-unit LSTMs, 384M parameters; a 5-model ensemble with beam 12 scored 34.81 BLEU on WMT'14 En→Fr vs 33.30 for phrase-based SMT.
← back to Deep Learning roadmap