# Machine translation

> Map a sentence in one language to an equivalent one in another: hand-written rules, then statistical alignment and phrase tables, now neural encoder–decoders, scored against references with BLEU, chrF or COMET.

Source: https://www.mlfoundry.in/learn/machine-translation  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**Machine translation (MT):** is the task of mapping a source-language text $x=(x_1,\dots,x_S)$ to a target-language text $y=(y_1,\dots,y_T)$ with the same meaning. A system is judged on **adequacy** (is the meaning preserved?) and **fluency** (is it natural target-language text?), usually by automatic comparison with one or more human reference translations.

The input and output are both sequences, and their lengths differ. Words reorder: German puts the verb at the end of a subordinate clause, Japanese puts it last in every clause. One word can become several (German *Kopfschmerztablette* is *headache tablet*, and *Schadenfreude* needs a whole phrase), and many translations are correct, so there is no single right answer to train toward or test against.

Ambiguity has to be resolved from context. English *bank* is *Bank* or *Ufer* in German. Translating English *you* into French means choosing between *tu* and *vous*, which depends on who is speaking to whom, often stated sentences earlier.

## Three generations of systems

**Rule-based MT** (1950s–1980s) used linguists' grammars and dictionaries: parse the source, transfer the structure with hand-written rules, generate the target. The Georgetown–IBM demonstration of 1954 translated a few dozen Russian sentences into English. The 1966 ALPAC report concluded that MT was slower, less accurate and more expensive than human translation, and US funding largely stopped.

**Statistical MT** learned everything from parallel text. Brown et al. (1993, the IBM Models 1–5) wrote translation as a noisy channel, $\hat e=\arg\max_e P(e)\,P(f\mid e)$, and learned word-alignment probabilities with EM. **Phrase-based** SMT (Koehn, Och & Marcu 2003) translated contiguous word groups instead of single words, and Och & Ney (2002) replaced the two-factor product by a weighted log-linear model:

$$
\hat e=\arg\max_{e}\;\sum_{k=1}^{K}\lambda_k\,h_k(e,f)
$$

- $h_k(e,f)$: feature functions: phrase translation log-probabilities in both directions, a target n-gram language model $\log P(e)$, a distortion (reordering) cost, word and phrase counts
- $\lambda_k$: feature weights, tuned to maximise BLEU on a development set (minimum error rate training, Och 2003)
- $\arg\max_e$: found approximately by a beam-search decoder over phrase segmentations and orderings (e.g. Moses, 2007)

**Neural MT** (2014 onward) learns one network for $p_\theta(y\mid x)=\prod_t p_\theta(y_t\mid y_{<t},x)$ end to end. A pipeline of separately trained components becomes a single model. Google moved its production system to neural MT in 2016 (Wu et al., GNMT). The architecture, training and decoding details are in **Neural machine translation**.

> Figure: One German sentence through the three generations. Rule-based systems analyse, transfer with hand-written rules and generate. SMT learns word alignments with EM, extracts a phrase table (auf der Matte → on the mat) and searches for the best weighted sum of feature scores. NMT encodes subwords, attends to source states at every step and decodes token by token. All three are scored the same way, by comparing the output with human references.

## Measuring quality: BLEU

Human evaluation is the ground truth, but it is slow and expensive. **BLEU** (Papineni, Roukos, Ward & Zhu 2002) measures how many of the hypothesis's word n-grams appear in the references. Two rules stop it from being gamed. Each n-gram count is **clipped** to its maximum count in any one reference. A **brevity penalty** punishes output shorter than the references, since short output is the easy way to high precision.

$$
p_n=\frac{\sum_{g\in\text{n-grams}(\hat y)}\min\!\big(\mathrm{cnt}_{\hat y}(g),\,\max_{r}\mathrm{cnt}_{r}(g)\big)}{\sum_{g}\mathrm{cnt}_{\hat y}(g)},\qquad \mathrm{BLEU}=\mathrm{BP}\cdot\exp\Big(\tfrac1N\sum_{n=1}^{N}\log p_n\Big),\qquad \mathrm{BP}=\begin{cases}1&c>r\\ e^{\,1-r/c}&c\le r\end{cases}
$$

- $p_n$: clipped (modified) n-gram precision; standard BLEU uses $N=4$ with equal weights
- $c,\;r$: hypothesis length and effective reference length (the closest reference length, summed over the corpus)
- $\mathrm{BP}$: brevity penalty; there is no recall term, so this is the only thing stopping short output

BLEU is defined at the **corpus** level: matches, totals and lengths are summed over all test sentences before the precisions are taken. At the sentence level one missing 4-gram makes $\log p_4=-\infty$ and BLEU 0, so sentence BLEU needs smoothing.

**chrF** (Popović 2015) uses character n-grams ($n\le6$), which gives partial credit for morphology (*sits* vs *sitting*). It combines precision $\mathrm{chrP}$ and recall $\mathrm{chrR}$, averaged over $n$, into an F-score with $\beta=2$, so recall counts twice as much:

$$
\mathrm{chrF}_\beta=(1+\beta^2)\,\frac{\mathrm{chrP}\cdot\mathrm{chrR}}{\beta^2\,\mathrm{chrP}+\mathrm{chrR}},\qquad\beta=2
$$

- $\mathrm{chrP},\,\mathrm{chrR}$: character n-gram precision and recall (spaces removed), averaged over n = 1…6

**COMET** (Rei et al. 2020) is a learned metric. A pretrained multilingual encoder reads the source, the hypothesis and the reference, and a regressor trained on human quality judgements predicts a score. It credits paraphrases that share no n-grams with the reference.

**Benchmarks and metric conventions**

- **WMT:** Annual shared tasks on news (now general-domain) translation, a workshop from 2006 and the Conference on Machine Translation from 2016; test sets such as newstest2014 are the standard benchmarks
- **WMT'14 En→De:** About 4.5M sentence pairs of training data; the Transformer (big) reached 28.4 BLEU on newstest2014 (Vaswani et al. 2017)
- **WMT'14 En→Fr:** About 36M sentence pairs; Transformer (big) 41.8 BLEU
- **sacreBLEU:** Post (2018): BLEU scores depend on tokenisation and casing, so report scores from one fixed implementation with its signature string
- **Metrics research:** The WMT22 metrics task report was titled “Stop Using BLEU – Neural Metrics Are Better and More Robust”

> Figure: Six English candidates for one German sentence, scored live against one or two references. Fluent candidate A has every order matched and BLEU 91.47 (chrF 90.59). Literal B gets 9/10, 6/9, 3/8 and 1/7, and is one word shorter than the closest reference, so BP = 0.9048 and BLEU = 38.31. Paraphrase C has no 3-gram match, so BLEU is 0. Add-one smoothing gives it 17.47. Too-short D has BP = exp(1 − 11/4) = 0.1738; turn BP off and its smoothed score jumps from 8.69 to 50.00. Word salad E has p₁ = 12/12 yet BLEU 0, while chrF still gives it 48.97, double C's 24.29. Drop to one reference and A falls to 83.52.

## Worked example: BLEU of the literal candidate

Hypothesis *the cat sits since the morning on the warm mat* (10 words). References: *the cat has been sitting on the warm mat since the morning* (12 words) and *since this morning the cat has sat on the warm mat* (11 words).

Clipped n-gram counts against both references (the figure's candidate B)

| n | Matched n-grams | p_n | log p_n |
| --- | --- | --- | --- |
| 1 | the ×3 (clipped at 3), cat, since, morning, on, warm, mat; not sits | 9/10 = 0.900 | −0.1054 |
| 2 | the cat, since the, the morning, on the, the warm, warm mat | 6/9 = 0.667 | −0.4055 |
| 3 | since the morning, on the warm, the warm mat | 3/8 = 0.375 | −0.9808 |
| 4 | on the warm mat | 1/7 = 0.143 | −1.9459 |

1. Geometric mean: $\exp\big((-0.1054-0.4055-0.9808-1.9459)/4\big)=\exp(-0.8594)=0.4234$.
2. Lengths: $c=10$; the closest reference has $r=11$, so $\mathrm{BP}=e^{1-11/10}=e^{-0.1}=0.9048$.
3. $\mathrm{BLEU}=0.9048\times0.4234=0.3831$, reported as **38.31**.
4. Add the 4-word hypothesis *the cat is sitting* as a second segment and compute corpus BLEU: counts become 12/14, 7/12, 3/10, 1/8, lengths $c=14$, $r=22$, $\mathrm{BP}=0.5647$, BLEU **20.90**. That sentence has sentence BLEU 0, but pooled counts stay defined.

```python
import math
from collections import Counter

def ngrams(toks, n):
    return Counter(tuple(toks[i:i + n]) for i in range(len(toks) - n + 1))

def corpus_bleu(hyps, refs, N=4):
    """hyps: list of strings; refs: one list of reference strings per segment."""
    match, total, c_len, r_len = [0] * N, [0] * N, 0, 0
    for h, rs in zip(hyps, refs):
        h, rs = h.split(), [r.split() for r in rs]
        c_len += len(h)
        r_len += min((abs(len(r) - len(h)), len(r)) for r in rs)[1]   # closest ref length
        for n in range(1, N + 1):
            max_ref = Counter()
            for r in rs:
                max_ref |= ngrams(r, n)                                 # max count over refs
            match[n - 1] += sum((ngrams(h, n) & max_ref).values())       # clipped counts
            total[n - 1] += max(0, len(h) - n + 1)
    bp = 1.0 if c_len > r_len else math.exp(1 - r_len / c_len)
    log_p = sum(math.log(m / t) for m, t in zip(match, total)) / N
    return 100 * bp * math.exp(log_p), match, total, bp

refs = ["the cat has been sitting on the warm mat since the morning",
        "since this morning the cat has sat on the warm mat"]
hyps = ["the cat sits since the morning on the warm mat",   # sentence BLEU 38.31
        "the cat is sitting"]                              # sentence BLEU 0
print(corpus_bleu(hyps, [refs, refs]))
# (20.90..., [12, 7, 3, 1], [14, 12, 10, 8], 0.5647...)
```

Corpus BLEU from scratch (run; output in the comment). Use sacrebleu.corpus_bleu for reportable numbers

> **BLEU numbers are not portable:** Scores change by whole points with tokenisation (`13a` vs `intl`), casing, the number of references and whether the test set was pre-tokenised. Compare systems only with the same sacreBLEU signature on the same test set, and treat differences under about one BLEU point with suspicion unless a significance test supports them.

> **Precision with a length penalty, not recall:** With several references, recall is ill-defined: the hypothesis should not have to contain every reference's words. BLEU therefore measures precision only and uses the brevity penalty as a crude stand-in for recall. chrF and COMET handle multiple valid outputs differently, which is why they rank paraphrases better.

**Strengths**

- BLEU and chrF are fast, deterministic and language-independent: they need only reference text
- Corpus BLEU correlates reasonably with human rankings of very different systems
- COMET correlates better with human judgements, especially among strong neural systems

**Limitations**

- BLEU and chrF reward surface overlap: valid paraphrases score low and fluent word swaps can score high
- Sentence-level BLEU is noisy and needs smoothing
- COMET is itself a neural model: slower, opaque, and exploitable if a system is tuned against it

**Question:** Reference: “the cat is on the mat”. Hypothesis: “the the the the the the the”. What is the clipped unigram precision p₁?

- A. 7/7 = 1
- B. 2/7
- C. 1/7
- D. 0

**Answer:** B. 2/7 — Every hypothesis word is *the*, which appears in the reference, so unclipped precision would be 7/7. Clipping caps the count of *the* at its reference count, 2, giving $p_1=2/7$. This is Papineni et al.'s own example.

**Question:** A 4-word hypothesis has references of 11 and 12 words. What is its brevity penalty?

- A. 4/11 ≈ 0.364
- B. exp(1 − 11/4) ≈ 0.174
- C. exp(1 − 12/4) ≈ 0.135
- D. 1, because BP only applies at corpus level

**Answer:** B. exp(1 − 11/4) ≈ 0.174 — $r$ is the reference length closest to $c=4$, which is 11. Then $\mathrm{BP}=e^{1-11/4}=e^{-1.75}=0.1738$. That is candidate D in the figure.

**Key points**

- MT maps a source sentence to an equivalent target sentence. It is hard because of reordering, one-to-many word mappings, ambiguity, and many valid outputs.
- Rule-based (hand-written transfer) → SMT (IBM alignment models 1993, phrase-based 2003, log-linear $\sum\lambda_k h_k$) → neural $p_\theta(y\mid x)$ trained end to end.
- BLEU = BP · geometric mean of clipped 1–4-gram precisions; $\mathrm{BP}=e^{1-r/c}$ when $c\le r$; compute it at corpus level with sacreBLEU.
- chrF uses character n-grams and a recall-weighted F₂; COMET is a learned metric that correlates better with humans.
- Worked example: 9/10, 6/9, 3/8, 1/7 and BP 0.9048 give BLEU 38.31.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
