Definition
Machine translation (MT) is the task of mapping a source-language text to a target-language text with the same meaning. A system is judged on adequacy (is the meaning preserved?) and fluency (is it natural target-language text?), usually by automatic comparison with one or more human reference translations.
The input and output are both sequences, and their lengths differ. Words reorder: German puts the verb at the end of a subordinate clause, Japanese puts it last in every clause. One word can become several (German Kopfschmerztablette is headache tablet, and Schadenfreude needs a whole phrase), and many translations are correct, so there is no single right answer to train toward or test against.
Ambiguity has to be resolved from context. English bank is Bank or Ufer in German. Translating English you into French means choosing between tu and vous, which depends on who is speaking to whom, often stated sentences earlier.
Three generations of systems
Rule-based MT (1950s–1980s) used linguists' grammars and dictionaries: parse the source, transfer the structure with hand-written rules, generate the target. The Georgetown–IBM demonstration of 1954 translated a few dozen Russian sentences into English. The 1966 ALPAC report concluded that MT was slower, less accurate and more expensive than human translation, and US funding largely stopped.
Statistical MT learned everything from parallel text. Brown et al. (1993, the IBM Models 1–5) wrote translation as a noisy channel, , and learned word-alignment probabilities with EM. Phrase-based SMT (Koehn, Och & Marcu 2003) translated contiguous word groups instead of single words, and Och & Ney (2002) replaced the two-factor product by a weighted log-linear model:
- feature functions: phrase translation log-probabilities in both directions, a target n-gram language model , a distortion (reordering) cost, word and phrase counts
- feature weights, tuned to maximise BLEU on a development set (minimum error rate training, Och 2003)
- found approximately by a beam-search decoder over phrase segmentations and orderings (e.g. Moses, 2007)
Neural MT (2014 onward) learns one network for end to end. A pipeline of separately trained components becomes a single model. Google moved its production system to neural MT in 2016 (Wu et al., GNMT). The architecture, training and decoding details are in Neural machine translation.
Measuring quality: BLEU
Human evaluation is the ground truth, but it is slow and expensive. BLEU (Papineni, Roukos, Ward & Zhu 2002) measures how many of the hypothesis's word n-grams appear in the references. Two rules stop it from being gamed. Each n-gram count is clipped to its maximum count in any one reference. A brevity penalty punishes output shorter than the references, since short output is the easy way to high precision.
- clipped (modified) n-gram precision; standard BLEU uses with equal weights
- hypothesis length and effective reference length (the closest reference length, summed over the corpus)
- brevity penalty; there is no recall term, so this is the only thing stopping short output
BLEU is defined at the corpus level: matches, totals and lengths are summed over all test sentences before the precisions are taken. At the sentence level one missing 4-gram makes and BLEU 0, so sentence BLEU needs smoothing.
chrF (Popović 2015) uses character n-grams (), which gives partial credit for morphology (sits vs sitting). It combines precision and recall , averaged over , into an F-score with , so recall counts twice as much:
- character n-gram precision and recall (spaces removed), averaged over n = 1…6
COMET (Rei et al. 2020) is a learned metric. A pretrained multilingual encoder reads the source, the hypothesis and the reference, and a regressor trained on human quality judgements predicts a score. It credits paraphrases that share no n-grams with the reference.
Benchmarks and metric conventions
- WMT
- Annual shared tasks on news (now general-domain) translation, a workshop from 2006 and the Conference on Machine Translation from 2016; test sets such as newstest2014 are the standard benchmarks
- WMT'14 En→De
- About 4.5M sentence pairs of training data; the Transformer (big) reached 28.4 BLEU on newstest2014 (Vaswani et al. 2017)
- WMT'14 En→Fr
- About 36M sentence pairs; Transformer (big) 41.8 BLEU
- sacreBLEU
- Post (2018): BLEU scores depend on tokenisation and casing, so report scores from one fixed implementation with its signature string
- Metrics research
- The WMT22 metrics task report was titled “Stop Using BLEU – Neural Metrics Are Better and More Robust”
Worked example: BLEU of the literal candidate
Hypothesis the cat sits since the morning on the warm mat (10 words). References: the cat has been sitting on the warm mat since the morning (12 words) and since this morning the cat has sat on the warm mat (11 words).
| n | Matched n-grams | p_n | log p_n |
|---|---|---|---|
| 1 | the ×3 (clipped at 3), cat, since, morning, on, warm, mat; not sits | 9/10 = 0.900 | −0.1054 |
| 2 | the cat, since the, the morning, on the, the warm, warm mat | 6/9 = 0.667 | −0.4055 |
| 3 | since the morning, on the warm, the warm mat | 3/8 = 0.375 | −0.9808 |
| 4 | on the warm mat | 1/7 = 0.143 | −1.9459 |
- 1Geometric mean: .
- 2Lengths: ; the closest reference has , so .
- 3, reported as 38.31.
- 4Add the 4-word hypothesis the cat is sitting as a second segment and compute corpus BLEU: counts become 12/14, 7/12, 3/10, 1/8, lengths , , , BLEU 20.90. That sentence has sentence BLEU 0, but pooled counts stay defined.
1import math2from collections import Counter34def ngrams(toks, n):5 return Counter(tuple(toks[i:i + n]) for i in range(len(toks) - n + 1))67def corpus_bleu(hyps, refs, N=4):8 """hyps: list of strings; refs: one list of reference strings per segment."""9 match, total, c_len, r_len = [0] * N, [0] * N, 0, 010 for h, rs in zip(hyps, refs):11 h, rs = h.split(), [r.split() for r in rs]12 c_len += len(h)13 r_len += min((abs(len(r) - len(h)), len(r)) for r in rs)[1] # closest ref length14 for n in range(1, N + 1):15 max_ref = Counter()16 for r in rs:17 max_ref |= ngrams(r, n) # max count over refs18 match[n - 1] += sum((ngrams(h, n) & max_ref).values()) # clipped counts19 total[n - 1] += max(0, len(h) - n + 1)20 bp = 1.0 if c_len > r_len else math.exp(1 - r_len / c_len)21 log_p = sum(math.log(m / t) for m, t in zip(match, total)) / N22 return 100 * bp * math.exp(log_p), match, total, bp2324refs = ["the cat has been sitting on the warm mat since the morning",25 "since this morning the cat has sat on the warm mat"]26hyps = ["the cat sits since the morning on the warm mat", # sentence BLEU 38.3127 "the cat is sitting"] # sentence BLEU 028print(corpus_bleu(hyps, [refs, refs]))29# (20.90..., [12, 7, 3, 1], [14, 12, 10, 8], 0.5647...)Use when
- +BLEU and chrF are fast, deterministic and language-independent: they need only reference text
- +Corpus BLEU correlates reasonably with human rankings of very different systems
- +COMET correlates better with human judgements, especially among strong neural systems
Avoid when
- −BLEU and chrF reward surface overlap: valid paraphrases score low and fluent word swaps can score high
- −Sentence-level BLEU is noisy and needs smoothing
- −COMET is itself a neural model: slower, opaque, and exploitable if a system is tuned against it
Check yourself
Reference: “the cat is on the mat”. Hypothesis: “the the the the the the the”. What is the clipped unigram precision p₁?
Check yourself
A 4-word hypothesis has references of 11 and 12 words. What is its brevity penalty?
Remember
- ✓MT maps a source sentence to an equivalent target sentence. It is hard because of reordering, one-to-many word mappings, ambiguity, and many valid outputs.
- ✓Rule-based (hand-written transfer) → SMT (IBM alignment models 1993, phrase-based 2003, log-linear ) → neural trained end to end.
- ✓BLEU = BP · geometric mean of clipped 1–4-gram precisions; when ; compute it at corpus level with sacreBLEU.
- ✓chrF uses character n-grams and a recall-weighted F₂; COMET is a learned metric that correlates better with humans.
- ✓Worked example: 9/10, 6/9, 3/8, 1/7 and BP 0.9048 give BLEU 38.31.