Definition
Neural machine translation (NMT) trains a single network end to end on parallel sentences to model . An encoder reads the source and a decoder generates the target one subword at a time. The approach began with Kalchbrenner & Blunsom (2013), Sutskever, Vinyals & Le (2014) and Cho et al. (2014). It became practical with attention (Bahdanau, Cho & Bengio 2015) and became the standard with the Transformer (Vaswani et al. 2017).
This article covers the system-level recipe. The task, its history and its metrics are in Machine translation. The RNN encoder–decoder is in Sequence-to-sequence (Seq2Seq) and Bahdanau attention, and the decoding algorithm is in Beam search.
From one vector to attention to the Transformer
Sutskever et al. compressed the whole source into the final state of a 4-layer LSTM. With the source reversed and a 5-model ensemble this reached 34.8 BLEU on WMT'14 English→French. A fixed-size vector is a bottleneck, though, and quality fell on long sentences.
Bahdanau et al. kept every encoder state and let the decoder compute, at each step, a soft alignment over them: a weighted average of source states chosen by the current decoder state. The alignment that SMT learned as a separate model became a differentiable part of the network.
The Transformer dropped recurrence. The encoder uses self-attention over the source. Each decoder layer uses masked self-attention over the target prefix, then cross-attention whose queries come from the decoder and whose keys and values are the encoder outputs. Every position in a layer is computed in parallel during training, so the model trains far faster than an LSTM of similar quality.
Transformer configurations for NMT (Vaswani et al. 2017)
- Base
- 6 + 6 layers, , FFN 2048, 8 heads, 65M parameters; 27.3 BLEU on WMT'14 En→De, 38.1 on En→Fr
- Big
- , FFN 4096, 16 heads, 213M parameters; 28.4 BLEU En→De, 41.8 En→Fr
- Vocabulary
- Shared source–target byte-pair encoding of about 37,000 tokens for En–De; 32,000 word pieces for En–Fr. Input embeddings, output embeddings and the pre-softmax projection share one matrix
- Training
- Adam () with 4,000 warm-up steps, dropout 0.1 for base, label smoothing ; base 100k steps (12 hours on 8 P100 GPUs), big 300k steps (3.5 days)
- Decoding
- Beam size 4, length penalty
Subword vocabulary
A word-level softmax caps the vocabulary (often at 30k–80k words) and maps everything else to <unk>. That fails on names, numbers and compounds, which are exactly the words MT must copy or build. Sennrich, Haddow & Birch (2016) segmented words into subword units learned by Byte-pair encoding (BPE). Rare words are then spelled from frequent pieces, the vocabulary is open, and names can be transliterated piece by piece.
Learning one vocabulary on the concatenated source and target text lets identical strings (names, numbers, URLs) map to identical tokens. That is what makes tied embeddings work across languages. SentencePiece does the same on raw text without pre-tokenisation.
Training: label-smoothed cross-entropy
Training uses teacher forcing: the decoder sees the reference prefix and is scored on . Vaswani et al. replace the one-hot target with a smoothed one (Szegedy et al. 2016):
- smoothing mass, 0.1 in the Transformer paper; is ordinary maximum likelihood
- vocabulary size: the ε mass is spread uniformly over every token
- smoothed target. The loss equals , where is the cross-entropy against the uniform distribution
With the loss keeps falling as the reference logit grows, so training never stops pushing toward certainty. With the gradient on a logit is , which is zero at . The best achievable prediction then puts on the reference token. Vaswani et al. note that this hurts perplexity, since the model becomes less sure, but improves BLEU.
Decoding: beam search with length normalisation
Exact is intractable, so NMT keeps the best prefixes per step (see Beam search). Raw log-probability falls with every token added, which biases the beam toward short translations. The GNMT system (Wu et al. 2016) divides by a length penalty:
- 0 gives raw log-probability; 1 gives roughly per-token averaging; the Transformer used 0.6
- hypothesis length in tokens
Wider beams are not always better. Koehn & Knowles (2017) found that NMT quality drops for large beams. Stahlberg & Byrne (2019) showed why: with exact search, the model's single most probable output is often the empty translation, so a small beam's search errors partly hide the model's errors.
Back-translation: using monolingual data
Parallel text is scarce, but target-language monolingual text is plentiful. Sennrich, Haddow & Birch (2016) trained a reverse model (target→source) and used it to translate monolingual target sentences. Each output becomes a synthetic source paired with a real, human-written target. The decoder only ever learns to produce real text. The noise sits in the source, which the encoder learns to tolerate.
At scale, how the synthetic sources are generated matters. Edunov, Ott, Auli & Grangier (2018) found that sampling, or beam search with added noise, beats plain beam search, because beam output is too regular to teach much. They reached 35.0 BLEU on WMT'14 English→German.
| Quantity | Computation | Value |
|---|---|---|
| q(reference) | 0.900003 | |
| q(each other token) | ||
| Optimal margin m* | 12.571 | |
| Loss at m* = H(q) | 1.3624 | |
| NLL at m* | 0.1054 | |
| m = 20: smoothed loss vs NLL | overconfident logit | 2.0000 vs 0.0001 |
Worked example 2: short vs long hypothesis, log p = −2.4 (|y| = 3) vs −3.2 (|y| = 7)
- 1: scores vs , so the short, incomplete the cat . wins.
- 2: and , giving vs . The short one still wins narrowly.
- 3: and , giving vs . Now the full the cat sits on the mat . wins.
1import math23def smoothed_ce(m, V, eps):4 """Label-smoothed CE when the reference token has logit m and the other V-1 have 0."""5 lse = math.log(math.exp(m) + V - 1) # log softmax denominator6 logp_ref, logp_other = m - lse, -lse7 nll = -logp_ref8 uniform = -(logp_ref + (V - 1) * logp_other) / V # CE against the uniform distribution9 return (1 - eps) * nll + eps * uniform, nll1011V, eps = 32000, 0.112q = 1 - eps + eps / V13m_star = math.log(q * (V - 1) / (1 - q))14for m in [8.0, m_star, 16.0, 20.0]:15 loss, nll = smoothed_ce(m, V, eps)16 print(f"m={m:6.3f} smoothed={loss:.4f} nll={nll:.4f}")17# m= 8.000 smoothed=3.2625 nll=2.462518# m=12.571 smoothed=1.3624 nll=0.105419# m=16.000 smoothed=1.6035 nll=0.003620# m=20.000 smoothed=2.0000 nll=0.00012122# --- with a real model (requires torch + transformers) ---23# import torch.nn as nn24# from transformers import AutoTokenizer, AutoModelForSeq2SeqLM25# name = "Helsinki-NLP/opus-mt-de-en" # Marian Transformer trained on OPUS26# tok, model = AutoTokenizer.from_pretrained(name), AutoModelForSeq2SeqLM.from_pretrained(name)27# batch = tok(["Die Katze sitzt auf der Matte."], return_tensors="pt")28# labels = tok(text_target=["The cat sits on the mat."], return_tensors="pt").input_ids29# logits = model(**batch, labels=labels).logits # teacher forcing30# loss = nn.CrossEntropyLoss(label_smoothing=0.1)(logits.flatten(0, 1), labels.flatten())31# out = model.generate(**batch, num_beams=4, max_new_tokens=64)32# print(tok.batch_decode(out, skip_special_tokens=True))Check yourself
With label smoothing ε = 0.1 over V = 32,000 tokens, what probability on the reference token minimises the training loss at one step?
Check yourself
Why does back-translation translate target-language monolingual text into the source language, rather than the other way round?
Remember
- ✓NMT models with one network: from the LSTM encoder–decoder (2014) to attention (2015) to the Transformer (2017).
- ✓Transformer base: 6+6 layers, d = 512, 65M parameters, 27.3 BLEU En→De; big: d = 1024, 213M, 28.4 BLEU.
- ✓A shared subword vocabulary (BPE, about 37k) gives an open vocabulary and tied embeddings across both languages.
- ✓Label smoothing gives the loss a finite optimum; at , k, that is and . It hurts perplexity and helps BLEU.
- ✓Beam search (K = 4) divides by to undo the short-output bias; back-translation turns target monolingual text into extra training pairs.