Machine translation

Map a sentence in one language to an equivalent one in another: hand-written rules, then statistical alignment and phrase tables, now neural encoder–decoders, scored against references with BLEU, chrF or COMET.

8 min read795 wordscore0 viewsupdated 2026-10-15

Definition

Machine translation (MT) is the task of mapping a source-language text x=(x1,…,xS)x=(x_1,\dots,x_S) to a target-language text y=(y1,…,yT)y=(y_1,\dots,y_T) with the same meaning. A system is judged on adequacy (is the meaning preserved?) and fluency (is it natural target-language text?), usually by automatic comparison with one or more human reference translations.

The input and output are both sequences, and their lengths differ. Words reorder: German puts the verb at the end of a subordinate clause, Japanese puts it last in every clause. One word can become several (German Kopfschmerztablette is headache tablet, and Schadenfreude needs a whole phrase), and many translations are correct, so there is no single right answer to train toward or test against.

Ambiguity has to be resolved from context. English bank is Bank or Ufer in German. Translating English you into French means choosing between tu and vous, which depends on who is speaking to whom, often stated sentences earlier.

Three generations of systems

Rule-based MT (1950s–1980s) used linguists' grammars and dictionaries: parse the source, transfer the structure with hand-written rules, generate the target. The Georgetown–IBM demonstration of 1954 translated a few dozen Russian sentences into English. The 1966 ALPAC report concluded that MT was slower, less accurate and more expensive than human translation, and US funding largely stopped.

Statistical MT learned everything from parallel text. Brown et al. (1993, the IBM Models 1–5) wrote translation as a noisy channel, e^=arg⁡max⁡eP(e) P(f∣e)\hat e=\arg\max_e P(e)\,P(f\mid e), and learned word-alignment probabilities with EM. Phrase-based SMT (Koehn, Och & Marcu 2003) translated contiguous word groups instead of single words, and Och & Ney (2002) replaced the two-factor product by a weighted log-linear model:

e^=arg⁡max⁡e  ∑k=1Kλk hk(e,f)\hat e=\arg\max_{e}\;\sum_{k=1}^{K}\lambda_k\,h_k(e,f)
hk(e,f)h_k(e,f)
feature functions: phrase translation log-probabilities in both directions, a target n-gram language model log⁡P(e)\log P(e), a distortion (reordering) cost, word and phrase counts
λk\lambda_k
feature weights, tuned to maximise BLEU on a development set (minimum error rate training, Och 2003)
arg⁡max⁡e\arg\max_e
found approximately by a beam-search decoder over phrase segmentations and orderings (e.g. Moses, 2007)

Neural MT (2014 onward) learns one network for pθ(y∣x)=∏tpθ(yt∣y<t,x)p_\theta(y\mid x)=\prod_t p_\theta(y_t\mid y_{<t},x) end to end. A pipeline of separately trained components becomes a single model. Google moved its production system to neural MT in 2016 (Wu et al., GNMT). The architecture, training and decoding details are in Neural machine translation.

Die Katze sitzt auf der Matte. → the cat sits on the mat.the same input through three generations of systemsrule-based1950s–1980sanalysis: parse GermanKatze = noun, fem., nom.transfer: hand-written rules+ bilingual dictionarygeneration: Englishword order + morphologyGeorgetown–IBM demo 1954statistical (SMT)1990–2016IBM Models 1–5: wordalignments learned by EMphrase table:auf der Matte → on the matlog-linear decoder: TM +LM + reordering + lengthBrown 1993; Koehn 2003neural (NMT)2014–encoder: subword tokens→ contextual vectorsattention: soft alignmentto source states per stepdecoder: p(y_t | y_<t, x)→ beam searchBahdanau 2015; Vaswani 2017ê = argmax_e Σ λ_k h_k(e, f)θ* = argmax Σ log p_θ(y | x)no learning: rules by linguistsevaluation: hypothesis vs human reference(s), on WMT test sets (shared tasks since 2006)BLEU (2002)clipped word 1–4-gramprecision × brevity penaltychrF (2015)character 1–6-gram F-score,β = 2 weights recallCOMET (2020)neural regressor on source,hypothesis, ref → human score
fig.One German sentence through the three generations. Rule-based systems analyse, transfer with hand-written rules and generate. SMT learns word alignments with EM, extracts a phrase table (auf der Matte → on the mat) and searches for the best weighted sum of feature scores. NMT encodes subwords, attends to source states at every step and decodes token by token. All three are scored the same way, by comparing the output with human references.

Measuring quality: BLEU

Human evaluation is the ground truth, but it is slow and expensive. BLEU (Papineni, Roukos, Ward & Zhu 2002) measures how many of the hypothesis's word n-grams appear in the references. Two rules stop it from being gamed. Each n-gram count is clipped to its maximum count in any one reference. A brevity penalty punishes output shorter than the references, since short output is the easy way to high precision.

pn=∑g∈n-grams(y^)min⁡ ⁣(cnty^(g), max⁡rcntr(g))∑gcnty^(g),BLEU=BP⋅exp⁡(1N∑n=1Nlog⁡pn),BP={1c>re 1−r/cc≤rp_n=\frac{\sum_{g\in\text{n-grams}(\hat y)}\min\!\big(\mathrm{cnt}_{\hat y}(g),\,\max_{r}\mathrm{cnt}_{r}(g)\big)}{\sum_{g}\mathrm{cnt}_{\hat y}(g)},\qquad \mathrm{BLEU}=\mathrm{BP}\cdot\exp\Big(\tfrac1N\sum_{n=1}^{N}\log p_n\Big),\qquad \mathrm{BP}=\begin{cases}1&c>r\\ e^{\,1-r/c}&c\le r\end{cases}
pnp_n
clipped (modified) n-gram precision; standard BLEU uses N=4N=4 with equal weights
c,  rc,\;r
hypothesis length and effective reference length (the closest reference length, summed over the corpus)
BP\mathrm{BP}
brevity penalty; there is no recall term, so this is the only thing stopping short output

BLEU is defined at the corpus level: matches, totals and lengths are summed over all test sentences before the precisions are taken. At the sentence level one missing 4-gram makes log⁡p4=−∞\log p_4=-\infty and BLEU 0, so sentence BLEU needs smoothing.

chrF (Popović 2015) uses character n-grams (n≤6n\le6), which gives partial credit for morphology (sits vs sitting). It combines precision chrP\mathrm{chrP} and recall chrR\mathrm{chrR}, averaged over nn, into an F-score with β=2\beta=2, so recall counts twice as much:

chrFβ=(1+β2) chrP⋅chrRβ2 chrP+chrR,β=2\mathrm{chrF}_\beta=(1+\beta^2)\,\frac{\mathrm{chrP}\cdot\mathrm{chrR}}{\beta^2\,\mathrm{chrP}+\mathrm{chrR}},\qquad\beta=2
chrP, chrR\mathrm{chrP},\,\mathrm{chrR}
character n-gram precision and recall (spaces removed), averaged over n = 1…6

COMET (Rei et al. 2020) is a learned metric. A pretrained multilingual encoder reads the source, the hypothesis and the reference, and a regressor trained on human quality judgements predicts a score. It credits paraphrases that share no n-grams with the reference.

Benchmarks and metric conventions

WMT
Annual shared tasks on news (now general-domain) translation, a workshop from 2006 and the Conference on Machine Translation from 2016; test sets such as newstest2014 are the standard benchmarks
WMT'14 En→De
About 4.5M sentence pairs of training data; the Transformer (big) reached 28.4 BLEU on newstest2014 (Vaswani et al. 2017)
WMT'14 En→Fr
About 36M sentence pairs; Transformer (big) 41.8 BLEU
sacreBLEU
Post (2018): BLEU scores depend on tokenisation and casing, so report scores from one fixed implementation with its signature string
Metrics research
The WMT22 metrics task report was titled “Stop Using BLEU – Neural Metrics Are Better and More Robust”
source (de): Die Katze sitzt seit dem Morgen auf der warmen Matte.ref 1 (12 w): the cat has been sitting on the warm mat since the morningref 2 (11 w): since this morning the cat has sat on the warm matcandidate A (fluent, 11 words) · green = clipped unigram matchthecathasbeensittingonthewarmmatsincemorningclipped n-gram precisionp111/11 = 1.000p29/10 = 0.900p38/9 = 0.889p47/8 = 0.875c = 11 words, closest reference r = 11BP = exp(1 − 11/11) = 1.0000BLEU-4 = BP · (Π p_n)^(1/4) = 91.47chrF (char 1–6-grams, β = 2) = 90.59every order has a match: the geometric mean is definedall candidates: BLEU · chrFA fluent91.590.6B literal38.368.6C paraphrase0.024.3D too short0.021.5E word salad0.049.0F repetition0.011.6Sentence BLEU is noisy: one missing 4-gram can zero it. Report corpus BLEU (sum matches andlengths over the whole test set first) with sacreBLEU. With both references the adequateparaphrase C scores no higher than word salad E on BLEU or chrF; learned COMET targets this.
candidate
referencessmoothingbrevity penalty
fig.Six English candidates for one German sentence, scored live against one or two references. Fluent candidate A has every order matched and BLEU 91.47 (chrF 90.59). Literal B gets 9/10, 6/9, 3/8 and 1/7, and is one word shorter than the closest reference, so BP = 0.9048 and BLEU = 38.31. Paraphrase C has no 3-gram match, so BLEU is 0. Add-one smoothing gives it 17.47. Too-short D has BP = exp(1 − 11/4) = 0.1738; turn BP off and its smoothed score jumps from 8.69 to 50.00. Word salad E has p₁ = 12/12 yet BLEU 0, while chrF still gives it 48.97, double C's 24.29. Drop to one reference and A falls to 83.52.

Worked example: BLEU of the literal candidate

Hypothesis the cat sits since the morning on the warm mat (10 words). References: the cat has been sitting on the warm mat since the morning (12 words) and since this morning the cat has sat on the warm mat (11 words).

nMatched n-gramsp_nlog p_n
1the ×3 (clipped at 3), cat, since, morning, on, warm, mat; not sits9/10 = 0.900−0.1054
2the cat, since the, the morning, on the, the warm, warm mat6/9 = 0.667−0.4055
3since the morning, on the warm, the warm mat3/8 = 0.375−0.9808
4on the warm mat1/7 = 0.143−1.9459
Clipped n-gram counts against both references (the figure's candidate B)
  1. 1Geometric mean: exp⁡((−0.1054−0.4055−0.9808−1.9459)/4)=exp⁡(−0.8594)=0.4234\exp\big((-0.1054-0.4055-0.9808-1.9459)/4\big)=\exp(-0.8594)=0.4234.
  2. 2Lengths: c=10c=10; the closest reference has r=11r=11, so BP=e1−11/10=e−0.1=0.9048\mathrm{BP}=e^{1-11/10}=e^{-0.1}=0.9048.
  3. 3BLEU=0.9048×0.4234=0.3831\mathrm{BLEU}=0.9048\times0.4234=0.3831, reported as 38.31.
  4. 4Add the 4-word hypothesis the cat is sitting as a second segment and compute corpus BLEU: counts become 12/14, 7/12, 3/10, 1/8, lengths c=14c=14, r=22r=22, BP=0.5647\mathrm{BP}=0.5647, BLEU 20.90. That sentence has sentence BLEU 0, but pooled counts stay defined.
python
1import math
2from collections import Counter
3
4def ngrams(toks, n):
5 return Counter(tuple(toks[i:i + n]) for i in range(len(toks) - n + 1))
6
7def corpus_bleu(hyps, refs, N=4):
8 """hyps: list of strings; refs: one list of reference strings per segment."""
9 match, total, c_len, r_len = [0] * N, [0] * N, 0, 0
10 for h, rs in zip(hyps, refs):
11 h, rs = h.split(), [r.split() for r in rs]
12 c_len += len(h)
13 r_len += min((abs(len(r) - len(h)), len(r)) for r in rs)[1] # closest ref length
14 for n in range(1, N + 1):
15 max_ref = Counter()
16 for r in rs:
17 max_ref |= ngrams(r, n) # max count over refs
18 match[n - 1] += sum((ngrams(h, n) & max_ref).values()) # clipped counts
19 total[n - 1] += max(0, len(h) - n + 1)
20 bp = 1.0 if c_len > r_len else math.exp(1 - r_len / c_len)
21 log_p = sum(math.log(m / t) for m, t in zip(match, total)) / N
22 return 100 * bp * math.exp(log_p), match, total, bp
23
24refs = ["the cat has been sitting on the warm mat since the morning",
25 "since this morning the cat has sat on the warm mat"]
26hyps = ["the cat sits since the morning on the warm mat", # sentence BLEU 38.31
27 "the cat is sitting"] # sentence BLEU 0
28print(corpus_bleu(hyps, [refs, refs]))
29# (20.90..., [12, 7, 3, 1], [14, 12, 10, 8], 0.5647...)
Corpus BLEU from scratch (run; output in the comment). Use sacrebleu.corpus_bleu for reportable numbers

Use when

  • +BLEU and chrF are fast, deterministic and language-independent: they need only reference text
  • +Corpus BLEU correlates reasonably with human rankings of very different systems
  • +COMET correlates better with human judgements, especially among strong neural systems

Avoid when

  • −BLEU and chrF reward surface overlap: valid paraphrases score low and fluent word swaps can score high
  • −Sentence-level BLEU is noisy and needs smoothing
  • −COMET is itself a neural model: slower, opaque, and exploitable if a system is tuned against it

Check yourself

Reference: “the cat is on the mat”. Hypothesis: “the the the the the the the”. What is the clipped unigram precision p₁?

Check yourself

A 4-word hypothesis has references of 11 and 12 words. What is its brevity penalty?

Remember

  • ✓MT maps a source sentence to an equivalent target sentence. It is hard because of reordering, one-to-many word mappings, ambiguity, and many valid outputs.
  • ✓Rule-based (hand-written transfer) → SMT (IBM alignment models 1993, phrase-based 2003, log-linear ∑λkhk\sum\lambda_k h_k) → neural pθ(y∣x)p_\theta(y\mid x) trained end to end.
  • ✓BLEU = BP · geometric mean of clipped 1–4-gram precisions; BP=e1−r/c\mathrm{BP}=e^{1-r/c} when c≤rc\le r; compute it at corpus level with sacreBLEU.
  • ✓chrF uses character n-grams and a recall-weighted F₂; COMET is a learned metric that correlates better with humans.
  • ✓Worked example: 9/10, 6/9, 3/8, 1/7 and BP 0.9048 give BLEU 38.31.
← back to Deep Learning roadmap