Neural machine translation

One network models p(y|x) token by token: a Transformer encoder–decoder over a shared subword vocabulary, trained with label-smoothed cross-entropy, decoded by length-normalised beam search, boosted by back-translation.

8 min read895 wordsadvanced0 viewsupdated 2026-10-15

Definition

Neural machine translation (NMT) trains a single network end to end on parallel sentences to model pθ(y∣x)=∏t=1Tpθ(yt∣y<t,x)p_\theta(y\mid x)=\prod_{t=1}^{T}p_\theta(y_t\mid y_{<t},x). An encoder reads the source and a decoder generates the target one subword at a time. The approach began with Kalchbrenner & Blunsom (2013), Sutskever, Vinyals & Le (2014) and Cho et al. (2014). It became practical with attention (Bahdanau, Cho & Bengio 2015) and became the standard with the Transformer (Vaswani et al. 2017).

This article covers the system-level recipe. The task, its history and its metrics are in Machine translation. The RNN encoder–decoder is in Sequence-to-sequence (Seq2Seq) and Bahdanau attention, and the decoding algorithm is in Beam search.

From one vector to attention to the Transformer

Sutskever et al. compressed the whole source into the final state of a 4-layer LSTM. With the source reversed and a 5-model ensemble this reached 34.8 BLEU on WMT'14 English→French. A fixed-size vector is a bottleneck, though, and quality fell on long sentences.

Bahdanau et al. kept every encoder state and let the decoder compute, at each step, a soft alignment over them: a weighted average of source states chosen by the current decoder state. The alignment that SMT learned as a separate model became a differentiable part of the network.

The Transformer dropped recurrence. The encoder uses self-attention over the source. Each decoder layer uses masked self-attention over the target prefix, then cross-attention whose queries come from the decoder and whose keys and values are the encoder outputs. Every position in a layer is computed in parallel during training, so the model trains far faster than an LSTM of similar quality.

inference: beam search, K = 4score = log p(y|x) / ((5+|y|)/6)^0.6▁the▁cat▁sits▁on▁the▁mat</s>label-smoothed targets: 0.9·onehot + 0.1/Vlinear + softmax over V (tied with embeddings)encoder × 6self-attention (8 heads) → FFN 2048sees the whole source at onced_model = 512 (base) · 1024 (big)decoder × 6masked self-attn → cross-attn → FFNtoken t sees y_<t and every source stateteacher forcing in trainingK, V▁Die▁Katze▁sitzt▁auf▁der▁Matte<s>▁the▁cat▁sits▁on▁the▁matsource: Die Katze sitzt auf der Mattetarget shifted right by oneone shared subword vocabulary for both languages (37k BPE for WMT'14 En–De) + positionsback-translation (more data)English monolingual text“the dog sleeps on the sofa”reverse model En→Debeam, sampling or noised beamsynthetic German source“der Hund schläft auf dem Sofa”pair (synthetic De, real En)training (Vaswani et al. 2017): Adam β₂ = 0.98, 4k warm-up steps, dropout 0.1 (base), ε_ls = 0.1base: 100k steps, 12 h on 8 P100 GPUs · big: 300k steps, 3.5 days
fig.A Transformer NMT system translating German to English. Source subwords (the split of Matte into ▁Mat + te is illustrative) enter a 6-layer encoder. The decoder reads the target shifted right by one and attends to the encoder states through cross-attention. A softmax over the shared vocabulary is trained against label-smoothed targets and decoded with a beam of 4 and the GNMT length penalty. Top left: back-translation, where a reverse En→De model turns English monolingual text into synthetic German sources, giving extra (synthetic De, real En) training pairs.

Transformer configurations for NMT (Vaswani et al. 2017)

Base
6 + 6 layers, dmodel=512d_\text{model}=512, FFN 2048, 8 heads, 65M parameters; 27.3 BLEU on WMT'14 En→De, 38.1 on En→Fr
Big
dmodel=1024d_\text{model}=1024, FFN 4096, 16 heads, 213M parameters; 28.4 BLEU En→De, 41.8 En→Fr
Vocabulary
Shared source–target byte-pair encoding of about 37,000 tokens for En–De; 32,000 word pieces for En–Fr. Input embeddings, output embeddings and the pre-softmax projection share one matrix
Training
Adam (β2=0.98\beta_2=0.98) with 4,000 warm-up steps, dropout 0.1 for base, label smoothing εls=0.1\varepsilon_{ls}=0.1; base 100k steps (12 hours on 8 P100 GPUs), big 300k steps (3.5 days)
Decoding
Beam size 4, length penalty α=0.6\alpha=0.6

Subword vocabulary

A word-level softmax caps the vocabulary (often at 30k–80k words) and maps everything else to <unk>. That fails on names, numbers and compounds, which are exactly the words MT must copy or build. Sennrich, Haddow & Birch (2016) segmented words into subword units learned by Byte-pair encoding (BPE). Rare words are then spelled from frequent pieces, the vocabulary is open, and names can be transliterated piece by piece.

Learning one vocabulary on the concatenated source and target text lets identical strings (names, numbers, URLs) map to identical tokens. That is what makes tied embeddings work across languages. SentencePiece does the same on raw text without pre-tokenisation.

Training: label-smoothed cross-entropy

Training uses teacher forcing: the decoder sees the reference prefix y<ty_{<t} and is scored on yty_t. Vaswani et al. replace the one-hot target with a smoothed one (Szegedy et al. 2016):

L(θ)=−∑t=1T∑k=1Vqt,k log⁡pθ(k∣y<t,x),qt,k=(1−ε) [k=yt]+εV\mathcal{L}(\theta)=-\sum_{t=1}^{T}\sum_{k=1}^{V}q_{t,k}\,\log p_\theta(k\mid y_{<t},x),\qquad q_{t,k}=(1-\varepsilon)\,[k=y_t]+\frac{\varepsilon}{V}
ε\varepsilon
smoothing mass, 0.1 in the Transformer paper; ε=0\varepsilon=0 is ordinary maximum likelihood
VV
vocabulary size: the ε mass is spread uniformly over every token
qt,kq_{t,k}
smoothed target. The loss equals (1−ε) NLL+ε U(1-\varepsilon)\,\mathrm{NLL}+\varepsilon\,U, where U=−1V∑klog⁡pkU=-\tfrac1V\sum_k\log p_k is the cross-entropy against the uniform distribution

With ε=0\varepsilon=0 the loss keeps falling as the reference logit grows, so training never stops pushing toward certainty. With ε>0\varepsilon>0 the gradient on a logit is pk−qkp_k-q_k, which is zero at p=qp=q. The best achievable prediction then puts 1−ε+ε/V1-\varepsilon+\varepsilon/V on the reference token. Vaswani et al. note that this hurts perplexity, since the model becomes less sure, but improves BLEU.

one decoder step: reference token “▁sits”, logit m vs 0 for the other 31,999048121620240246810logit margin m of the reference tokennatsm* = 12.57ε = 0.10ε = 0 (plain NLL)at m = 8.0p(ref)0.0852q(ref)0.9000p(other)2.86e-5q(other)3.13e-6NLL −log p2.4625U (uniform CE)10.4623loss (1−ε)NLL+εU3.2625∂loss/∂m = p − q-0.8148optimumm* = log(q(V−1)/(1−q)) = 12.571p(ref) = q = 0.9000min loss = H(q) = 1.3624NLL there = 0.1054 (not 0)below m*: both losses push the logit up
vocabulary V
fig.Loss at one decoder step when the reference token ▁sits has logit m and all other tokens 0. At the defaults (V = 32,000, ε = 0.1, m = 8) the model gives the reference only p = 0.0852, the loss is 3.2625 (plain NLL 2.4625) and the gradient −0.8148 pushes m up. The smoothed loss bottoms out at m* = 12.571, where p = 0.9000 and loss = H(q) = 1.3624. NLL there is 0.1054, not 0, which is the perplexity cost. Slide m to 16 and the gradient flips to +0.0964: overconfidence is now penalised. Set ε = 0 and the dashed curve keeps falling (0.0001 at m = 20) with no minimum. With V = 8, m* is only 4.29.

Decoding: beam search with length normalisation

Exact arg⁡max⁡yp(y∣x)\arg\max_y p(y\mid x) is intractable, so NMT keeps the KK best prefixes per step (see Beam search). Raw log-probability falls with every token added, which biases the beam toward short translations. The GNMT system (Wu et al. 2016) divides by a length penalty:

s(y)=log⁡pθ(y∣x)lp(y),lp(y)=(5+∣y∣)α(5+1)αs(y)=\frac{\log p_\theta(y\mid x)}{\mathrm{lp}(y)},\qquad \mathrm{lp}(y)=\frac{(5+|y|)^{\alpha}}{(5+1)^{\alpha}}
α\alpha
0 gives raw log-probability; 1 gives roughly per-token averaging; the Transformer used 0.6
∣y∣|y|
hypothesis length in tokens

Wider beams are not always better. Koehn & Knowles (2017) found that NMT quality drops for large beams. Stahlberg & Byrne (2019) showed why: with exact search, the model's single most probable output is often the empty translation, so a small beam's search errors partly hide the model's errors.

Back-translation: using monolingual data

Parallel text is scarce, but target-language monolingual text is plentiful. Sennrich, Haddow & Birch (2016) trained a reverse model (target→source) and used it to translate monolingual target sentences. Each output becomes a synthetic source paired with a real, human-written target. The decoder only ever learns to produce real text. The noise sits in the source, which the encoder learns to tolerate.

At scale, how the synthetic sources are generated matters. Edunov, Ott, Auli & Grangier (2018) found that sampling, or beam search with added noise, beats plain beam search, because beam output is too regular to teach much. They reached 35.0 BLEU on WMT'14 English→German.

QuantityComputationValue
q(reference)0.9+0.1/320000.9+0.1/320000.900003
q(each other token)0.1/320000.1/320003.125×10−63.125\times10^{-6}
Optimal margin m*log⁡(q(V−1)/(1−q))\log\big(q(V-1)/(1-q)\big)12.571
Loss at m* = H(q)(1−ε)NLL+εU(1-\varepsilon)\mathrm{NLL}+\varepsilon U1.3624
NLL at m*−log⁡0.900003-\log 0.9000030.1054
m = 20: smoothed loss vs NLLoverconfident logit2.0000 vs 0.0001
Worked example 1: smoothed targets at V = 32,000, ε = 0.1 (reference logit m, all others 0)

Worked example 2: short vs long hypothesis, log p = −2.4 (|y| = 3) vs −3.2 (|y| = 7)

  1. 1α=0\alpha=0: scores −2.4-2.4 vs −3.2-3.2, so the short, incomplete the cat . wins.
  2. 2α=0.6\alpha=0.6: lp(3)=(8/6)0.6=1.1884\mathrm{lp}(3)=(8/6)^{0.6}=1.1884 and lp(7)=(12/6)0.6=1.5157\mathrm{lp}(7)=(12/6)^{0.6}=1.5157, giving −2.0195-2.0195 vs −2.1112-2.1112. The short one still wins narrowly.
  3. 3α=1\alpha=1: lp=1.3333\mathrm{lp}=1.3333 and 2.02.0, giving −1.8-1.8 vs −1.6-1.6. Now the full the cat sits on the mat . wins.
python
1import math
2
3def smoothed_ce(m, V, eps):
4 """Label-smoothed CE when the reference token has logit m and the other V-1 have 0."""
5 lse = math.log(math.exp(m) + V - 1) # log softmax denominator
6 logp_ref, logp_other = m - lse, -lse
7 nll = -logp_ref
8 uniform = -(logp_ref + (V - 1) * logp_other) / V # CE against the uniform distribution
9 return (1 - eps) * nll + eps * uniform, nll
10
11V, eps = 32000, 0.1
12q = 1 - eps + eps / V
13m_star = math.log(q * (V - 1) / (1 - q))
14for m in [8.0, m_star, 16.0, 20.0]:
15 loss, nll = smoothed_ce(m, V, eps)
16 print(f"m={m:6.3f} smoothed={loss:.4f} nll={nll:.4f}")
17# m= 8.000 smoothed=3.2625 nll=2.4625
18# m=12.571 smoothed=1.3624 nll=0.1054
19# m=16.000 smoothed=1.6035 nll=0.0036
20# m=20.000 smoothed=2.0000 nll=0.0001
21
22# --- with a real model (requires torch + transformers) ---
23# import torch.nn as nn
24# from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
25# name = "Helsinki-NLP/opus-mt-de-en" # Marian Transformer trained on OPUS
26# tok, model = AutoTokenizer.from_pretrained(name), AutoModelForSeq2SeqLM.from_pretrained(name)
27# batch = tok(["Die Katze sitzt auf der Matte."], return_tensors="pt")
28# labels = tok(text_target=["The cat sits on the mat."], return_tensors="pt").input_ids
29# logits = model(**batch, labels=labels).logits # teacher forcing
30# loss = nn.CrossEntropyLoss(label_smoothing=0.1)(logits.flatten(0, 1), labels.flatten())
31# out = model.generate(**batch, num_beams=4, max_new_tokens=64)
32# print(tok.batch_decode(out, skip_special_tokens=True))
Top (run, output in comments): the figure's smoothed loss. Bottom (not run here): the same loss and beam decoding with a pretrained Marian NMT model

Check yourself

With label smoothing ε = 0.1 over V = 32,000 tokens, what probability on the reference token minimises the training loss at one step?

Check yourself

Why does back-translation translate target-language monolingual text into the source language, rather than the other way round?

Remember

  • ✓NMT models p(y∣x)=∏tp(yt∣y<t,x)p(y\mid x)=\prod_t p(y_t\mid y_{<t},x) with one network: from the LSTM encoder–decoder (2014) to attention (2015) to the Transformer (2017).
  • ✓Transformer base: 6+6 layers, d = 512, 65M parameters, 27.3 BLEU En→De; big: d = 1024, 213M, 28.4 BLEU.
  • ✓A shared subword vocabulary (BPE, about 37k) gives an open vocabulary and tied embeddings across both languages.
  • ✓Label smoothing q=(1−ε) onehot+ε/Vq=(1-\varepsilon)\,\text{onehot}+\varepsilon/V gives the loss a finite optimum; at ε=0.1\varepsilon=0.1, V=32V=32k, that is p=0.9p=0.9 and m∗=12.571m^*=12.571. It hurts perplexity and helps BLEU.
  • ✓Beam search (K = 4) divides by lp=((5+∣y∣)/6)α\mathrm{lp}=((5+|y|)/6)^{\alpha} to undo the short-output bias; back-translation turns target monolingual text into extra training pairs.
← back to Deep Learning roadmap