# Neural machine translation

> One network models p(y|x) token by token: a Transformer encoder–decoder over a shared subword vocabulary, trained with label-smoothed cross-entropy, decoded by length-normalised beam search, boosted by back-translation.

Source: https://www.mlfoundry.in/learn/neural-machine-translation  
Section: Deep Learning › Deep Learning for NLP  
Level: advanced · 8 min read · updated 2026-10-15

**Neural machine translation (NMT):** trains a single network end to end on parallel sentences to model $p_\theta(y\mid x)=\prod_{t=1}^{T}p_\theta(y_t\mid y_{<t},x)$. An encoder reads the source and a decoder generates the target one subword at a time. The approach began with Kalchbrenner & Blunsom (2013), Sutskever, Vinyals & Le (2014) and Cho et al. (2014). It became practical with attention (Bahdanau, Cho & Bengio 2015) and became the standard with the Transformer (Vaswani et al. 2017).

This article covers the system-level recipe. The task, its history and its metrics are in **Machine translation**. The RNN encoder–decoder is in **Sequence-to-sequence (Seq2Seq)** and **Bahdanau attention**, and the decoding algorithm is in **Beam search**.

## From one vector to attention to the Transformer

Sutskever et al. compressed the whole source into the final state of a 4-layer LSTM. With the source reversed and a 5-model ensemble this reached 34.8 BLEU on WMT'14 English→French. A fixed-size vector is a bottleneck, though, and quality fell on long sentences.

Bahdanau et al. kept every encoder state and let the decoder compute, at each step, a soft alignment over them: a weighted average of source states chosen by the current decoder state. The alignment that SMT learned as a separate model became a differentiable part of the network.

The Transformer dropped recurrence. The encoder uses self-attention over the source. Each decoder layer uses masked self-attention over the target prefix, then **cross-attention** whose queries come from the decoder and whose keys and values are the encoder outputs. Every position in a layer is computed in parallel during training, so the model trains far faster than an LSTM of similar quality.

> Figure: A Transformer NMT system translating German to English. Source subwords (the split of Matte into ▁Mat + te is illustrative) enter a 6-layer encoder. The decoder reads the target shifted right by one and attends to the encoder states through cross-attention. A softmax over the shared vocabulary is trained against label-smoothed targets and decoded with a beam of 4 and the GNMT length penalty. Top left: back-translation, where a reverse En→De model turns English monolingual text into synthetic German sources, giving extra (synthetic De, real En) training pairs.

**Transformer configurations for NMT (Vaswani et al. 2017)**

- **Base:** 6 + 6 layers, $d_\text{model}=512$, FFN 2048, 8 heads, 65M parameters; 27.3 BLEU on WMT'14 En→De, 38.1 on En→Fr
- **Big:** $d_\text{model}=1024$, FFN 4096, 16 heads, 213M parameters; 28.4 BLEU En→De, 41.8 En→Fr
- **Vocabulary:** Shared source–target byte-pair encoding of about 37,000 tokens for En–De; 32,000 word pieces for En–Fr. Input embeddings, output embeddings and the pre-softmax projection share one matrix
- **Training:** Adam ($\beta_2=0.98$) with 4,000 warm-up steps, dropout 0.1 for base, label smoothing $\varepsilon_{ls}=0.1$; base 100k steps (12 hours on 8 P100 GPUs), big 300k steps (3.5 days)
- **Decoding:** Beam size 4, length penalty $\alpha=0.6$

## Subword vocabulary

A word-level softmax caps the vocabulary (often at 30k–80k words) and maps everything else to `<unk>`. That fails on names, numbers and compounds, which are exactly the words MT must copy or build. Sennrich, Haddow & Birch (2016) segmented words into subword units learned by **Byte-pair encoding (BPE)**. Rare words are then spelled from frequent pieces, the vocabulary is open, and names can be transliterated piece by piece.

Learning one vocabulary on the concatenated source and target text lets identical strings (names, numbers, URLs) map to identical tokens. That is what makes tied embeddings work across languages. **SentencePiece** does the same on raw text without pre-tokenisation.

## Training: label-smoothed cross-entropy

Training uses teacher forcing: the decoder sees the reference prefix $y_{<t}$ and is scored on $y_t$. Vaswani et al. replace the one-hot target with a smoothed one (Szegedy et al. 2016):

$$
\mathcal{L}(\theta)=-\sum_{t=1}^{T}\sum_{k=1}^{V}q_{t,k}\,\log p_\theta(k\mid y_{<t},x),\qquad q_{t,k}=(1-\varepsilon)\,[k=y_t]+\frac{\varepsilon}{V}
$$

- $\varepsilon$: smoothing mass, 0.1 in the Transformer paper; $\varepsilon=0$ is ordinary maximum likelihood
- $V$: vocabulary size: the ε mass is spread uniformly over every token
- $q_{t,k}$: smoothed target. The loss equals $(1-\varepsilon)\,\mathrm{NLL}+\varepsilon\,U$, where $U=-\tfrac1V\sum_k\log p_k$ is the cross-entropy against the uniform distribution

With $\varepsilon=0$ the loss keeps falling as the reference logit grows, so training never stops pushing toward certainty. With $\varepsilon>0$ the gradient on a logit is $p_k-q_k$, which is zero at $p=q$. The best achievable prediction then puts $1-\varepsilon+\varepsilon/V$ on the reference token. Vaswani et al. note that this **hurts perplexity**, since the model becomes less sure, but **improves BLEU**.

> Figure: Loss at one decoder step when the reference token ▁sits has logit m and all other tokens 0. At the defaults (V = 32,000, ε = 0.1, m = 8) the model gives the reference only p = 0.0852, the loss is 3.2625 (plain NLL 2.4625) and the gradient −0.8148 pushes m up. The smoothed loss bottoms out at m* = 12.571, where p = 0.9000 and loss = H(q) = 1.3624. NLL there is 0.1054, not 0, which is the perplexity cost. Slide m to 16 and the gradient flips to +0.0964: overconfidence is now penalised. Set ε = 0 and the dashed curve keeps falling (0.0001 at m = 20) with no minimum. With V = 8, m* is only 4.29.

## Decoding: beam search with length normalisation

Exact $\arg\max_y p(y\mid x)$ is intractable, so NMT keeps the $K$ best prefixes per step (see **Beam search**). Raw log-probability falls with every token added, which biases the beam toward short translations. The GNMT system (Wu et al. 2016) divides by a length penalty:

$$
s(y)=\frac{\log p_\theta(y\mid x)}{\mathrm{lp}(y)},\qquad \mathrm{lp}(y)=\frac{(5+|y|)^{\alpha}}{(5+1)^{\alpha}}
$$

- $\alpha$: 0 gives raw log-probability; 1 gives roughly per-token averaging; the Transformer used 0.6
- $|y|$: hypothesis length in tokens

Wider beams are not always better. Koehn & Knowles (2017) found that NMT quality drops for large beams. Stahlberg & Byrne (2019) showed why: with exact search, the model's single most probable output is often the empty translation, so a small beam's search errors partly hide the model's errors.

## Back-translation: using monolingual data

Parallel text is scarce, but target-language monolingual text is plentiful. Sennrich, Haddow & Birch (2016) trained a reverse model (target→source) and used it to translate monolingual target sentences. Each output becomes a synthetic source paired with a real, human-written target. The decoder only ever learns to produce real text. The noise sits in the source, which the encoder learns to tolerate.

At scale, how the synthetic sources are generated matters. Edunov, Ott, Auli & Grangier (2018) found that sampling, or beam search with added noise, beats plain beam search, because beam output is too regular to teach much. They reached 35.0 BLEU on WMT'14 English→German.

Worked example 1: smoothed targets at V = 32,000, ε = 0.1 (reference logit m, all others 0)

| Quantity | Computation | Value |
| --- | --- | --- |
| q(reference) | $0.9+0.1/32000$ | 0.900003 |
| q(each other token) | $0.1/32000$ | $3.125\times10^{-6}$ |
| Optimal margin m* | $\log\big(q(V-1)/(1-q)\big)$ | 12.571 |
| Loss at m* = H(q) | $(1-\varepsilon)\mathrm{NLL}+\varepsilon U$ | 1.3624 |
| NLL at m* | $-\log 0.900003$ | 0.1054 |
| m = 20: smoothed loss vs NLL | overconfident logit | 2.0000 vs 0.0001 |

**Worked example 2: short vs long hypothesis, log p = −2.4 (|y| = 3) vs −3.2 (|y| = 7)**

1. $\alpha=0$: scores $-2.4$ vs $-3.2$, so the short, incomplete *the cat .* wins.
2. $\alpha=0.6$: $\mathrm{lp}(3)=(8/6)^{0.6}=1.1884$ and $\mathrm{lp}(7)=(12/6)^{0.6}=1.5157$, giving $-2.0195$ vs $-2.1112$. The short one still wins narrowly.
3. $\alpha=1$: $\mathrm{lp}=1.3333$ and $2.0$, giving $-1.8$ vs $-1.6$. Now the full *the cat sits on the mat .* wins.

```python
import math

def smoothed_ce(m, V, eps):
    """Label-smoothed CE when the reference token has logit m and the other V-1 have 0."""
    lse = math.log(math.exp(m) + V - 1)                  # log softmax denominator
    logp_ref, logp_other = m - lse, -lse
    nll = -logp_ref
    uniform = -(logp_ref + (V - 1) * logp_other) / V     # CE against the uniform distribution
    return (1 - eps) * nll + eps * uniform, nll

V, eps = 32000, 0.1
q = 1 - eps + eps / V
m_star = math.log(q * (V - 1) / (1 - q))
for m in [8.0, m_star, 16.0, 20.0]:
    loss, nll = smoothed_ce(m, V, eps)
    print(f"m={m:6.3f}  smoothed={loss:.4f}  nll={nll:.4f}")
# m= 8.000  smoothed=3.2625  nll=2.4625
# m=12.571  smoothed=1.3624  nll=0.1054
# m=16.000  smoothed=1.6035  nll=0.0036
# m=20.000  smoothed=2.0000  nll=0.0001

# --- with a real model (requires torch + transformers) ---
# import torch.nn as nn
# from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
# name = "Helsinki-NLP/opus-mt-de-en"                     # Marian Transformer trained on OPUS
# tok, model = AutoTokenizer.from_pretrained(name), AutoModelForSeq2SeqLM.from_pretrained(name)
# batch = tok(["Die Katze sitzt auf der Matte."], return_tensors="pt")
# labels = tok(text_target=["The cat sits on the mat."], return_tensors="pt").input_ids
# logits = model(**batch, labels=labels).logits           # teacher forcing
# loss = nn.CrossEntropyLoss(label_smoothing=0.1)(logits.flatten(0, 1), labels.flatten())
# out = model.generate(**batch, num_beams=4, max_new_tokens=64)
# print(tok.batch_decode(out, skip_special_tokens=True))
```

Top (run, output in comments): the figure's smoothed loss. Bottom (not run here): the same loss and beam decoding with a pretrained Marian NMT model

> **Not every length_penalty is GNMT's:** Hugging Face `generate(length_penalty=α)` divides the summed log-probability by $|y|^{\alpha}$, not by $((5+|y|)/6)^{\alpha}$, and its default is 1.0. A value tuned for one formula does not transfer to the other. Set it per model and per language pair on a development set.

> **Smoothing as calibration:** Label smoothing caps the logit margin, so softmax probabilities stay closer to the model's real accuracy. Müller, Kornblith & Hinton (2019) showed that it also pulls penultimate-layer representations of each class into tight, evenly spaced clusters. That helps BLEU, but it harms distillation, because the teacher's informative "dark knowledge" about similar tokens is erased.

**Question:** With label smoothing ε = 0.1 over V = 32,000 tokens, what probability on the reference token minimises the training loss at one step?

- A. 1.0
- B. 0.9 + 0.1/32,000 ≈ 0.900003
- C. 0.9 exactly
- D. 1/32,000

**Answer:** B. 0.9 + 0.1/32,000 ≈ 0.900003 — The loss is the cross-entropy $H(q,p)$, minimised at $p=q$, and the smoothed target puts $(1-\varepsilon)+\varepsilon/V$ on the reference. The figure shows the same optimum at margin $m^*=12.571$.

**Question:** Why does back-translation translate target-language monolingual text into the source language, rather than the other way round?

- A. The reverse model is always more accurate
- B. So the decoder is trained only on real, human-written targets; the synthetic noise sits in the encoder's input
- C. Source-language monolingual text cannot be tokenised with the shared vocabulary
- D. It avoids the need for a parallel corpus altogether

**Answer:** B. So the decoder is trained only on real, human-written targets; the synthetic noise sits in the encoder's input — The decoder learns to produce whatever appears on the target side. Keeping that side real protects output fluency, while the encoder learns to cope with imperfect synthetic sources. A parallel corpus is still needed to train the reverse model.

**Key points**

- NMT models $p(y\mid x)=\prod_t p(y_t\mid y_{<t},x)$ with one network: from the LSTM encoder–decoder (2014) to attention (2015) to the Transformer (2017).
- Transformer base: 6+6 layers, d = 512, 65M parameters, 27.3 BLEU En→De; big: d = 1024, 213M, 28.4 BLEU.
- A shared subword vocabulary (BPE, about 37k) gives an open vocabulary and tied embeddings across both languages.
- Label smoothing $q=(1-\varepsilon)\,\text{onehot}+\varepsilon/V$ gives the loss a finite optimum; at $\varepsilon=0.1$, $V=32$k, that is $p=0.9$ and $m^*=12.571$. It hurts perplexity and helps BLEU.
- Beam search (K = 4) divides by $\mathrm{lp}=((5+|y|)/6)^{\alpha}$ to undo the short-output bias; back-translation turns target monolingual text into extra training pairs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
