Multilingual models

One Transformer pre-trained on 100+ languages with one shared subword vocabulary and p^α language sampling (mBERT, XLM-R, mT5): fine-tune it in English and it transfers zero-shot to languages it never saw labels for.

8 min read682 wordsadvanced0 viewsupdated 2026-10-15

Definition

A multilingual model is a single network pre-trained on unlabelled text in many languages at once, with one shared subword vocabulary and no language-specific parameters. The main examples are mBERT (Devlin et al. 2018, 104 languages), XLM-R (Conneau et al. 2020, 100 languages) and mT5 (Xue et al. 2021, 101 languages). Its main use is zero-shot cross-lingual transfer: fine-tune on labelled data in one language, usually English, and apply the model to others.

Labelled data exists for a handful of languages, but users write in thousands. Before multilingual pre-training, the options were to translate the test input into English (translate-test), translate the English training set into each target language (translate-train), or align separately trained word embeddings with a bilingual dictionary. All of these need MT or dictionaries, and errors compound.

A multilingual model learns from monolingual text alone. mBERT and XLM-R use only masked language modelling on concatenated per-language corpora, with no parallel data and no language-ID input. Pires, Schlinger & Garrette (2019) still found that mBERT fine-tuned on one language works on others, even across scripts. Shared subwords (numbers, names, URLs), shared parameters, and the pressure to compress many languages into one space align their representations well enough to transfer.

raw text per language (GB)bar = share p · q at α = 0.3en30026.0%ru28025.5%de6616.5%ar2812.8%hi2011.5%sw1.65.4%yo0.12.4%temperature samplingq_i = p_i^α / Σ_j p_j^αone shared vocabularySentencePiece, 250k piecesone Transformer, MLMno language ID inputXLM-R: 100 languagesfine-tune: English onlyXNLI train = MultiNLIentail / neutral / contrazero-shot test14 other languages:fr es de el bg ru trar vi th zh hi sw urno target labels usedthe same sentence in six languages: the tokenizer's raw material differs by scriptlangsentencecharsUTF-8 byteswordsenThe cat sat on the mat.23236deDie Katze saß auf der Matte.28296swPaka aliketi juu ya mkeka.26265ruКошка сидела на коврике.24444hiबिल्ली चटाई पर बैठी थी।23615zh猫坐在垫子上。721– (no spaces)ASCII letters take 1 byte, Cyrillic 2, Devanagari and CJK 3. Fertility = subword tokens per word;it is lowest for languages that got many pieces in the shared vocabulary, highest for the rest.
fig.The pipeline. Per-language corpora are very unbalanced (illustrative sizes: English 300 GB, Yoruba 0.1 GB) and are resampled with q ∝ p^α, which at α = 0.3 lifts Yoruba from 0.01% to 2.4% of training data. One SentencePiece vocabulary of 250k pieces covers every script. One Transformer is pre-trained with masked language modelling, fine-tuned on English XNLI only, and tested on the 14 other XNLI languages. Below: the tokenizer's raw input differs by script. The same short sentence is 23 bytes in English, 44 in Russian and 61 in Hindi.

Balancing languages: exponent sampling

Web text is dominated by a few languages. Sampling training batches in proportion to corpus size starves low-resource languages. Sampling uniformly over languages repeats a 0.1 GB corpus hundreds of times while seeing a fraction of English. Every major model interpolates between the two:

qi=piα∑jpjα,pi=ni∑jnjq_i=\frac{p_i^{\alpha}}{\sum_{j}p_j^{\alpha}},\qquad p_i=\frac{n_i}{\sum_j n_j}
nin_i
amount of pre-training text in language i
qiq_i
probability that a training sequence (or batch) is drawn from language i
α\alpha
1 keeps natural proportions, 0 is uniform; mBERT smooths with exponent 0.7, XLM (Lample & Conneau 2019) uses 0.5, XLM-R and mT5 use 0.3

The same smoothing is usually applied when training the shared vocabulary. Otherwise high-resource scripts take most of the pieces, and other languages are split into many short fragments.

A language's fertility is the average number of subword tokens per word. High fertility means longer sequences, more compute per sentence, less text per context window, and pieces that carry less meaning each. Rust et al. (2021) found that languages with poorly covered vocabularies did worse in multilingual models than in monolingual ones.

fertility(ℓ)=# subword tokens of a text in ℓ# words of that text\text{fertility}(\ell)=\frac{\#\,\text{subword tokens of a text in }\ell}{\#\,\text{words of that text}}
ℓ\ell
a language; measured on the same (parallel) text across languages, fertility exposes how evenly the shared vocabulary serves them
q_i ∝ p_i^α with α = 0.30 · corpus sizes illustrative (GB)share (log scale) · ■ natural p · ■ sampled q0.01%0.1%1%10%100%en300 GB26.0%×0.6ru280 GB25.5%×0.6de66 GB16.5%×1.7ar28 GB12.8%×3.2hi20 GB11.5%×4.0sw1.6 GB5.41%×23.5yo0.1 GB2.35%×16400.250.50.751α (1 = natural, 0 = uniform)effective number of languages exp(H(q)) = 5.78 of 7 · English share 43.1% → 26.0%per 256-sequence batch: 66.5 English, 6.02 Yoruba (natural mix: 0.04)reading the corpus once in total, Yoruba is seen 163.7×, English 0.60×the compromise: low-resource languages upsampled, high-resource ones mildly downsampled
preset
fig.Seven languages with illustrative corpus sizes (695.7 GB in total). At the default α = 0.3 (XLM-R, mT5), English drops from 43.1% of the data to 26.0% and Yoruba rises from 0.01% to 2.35%. If training reads as much text as the corpus holds, Yoruba is seen 163.7 times and English 0.60 times, and a 256-sequence batch holds 6.02 Yoruba sequences instead of 0.04. The mixture is equivalent to 5.78 evenly used languages (exp of its entropy). The mBERT preset (0.7) gives Yoruba only 0.14% (×9.6). Uniform (α = 0) gives every language 14.29% and repeats Yoruba 993.9 times.

The curse of multilinguality

At a fixed model size, adding languages helps low-resource ones at first, through transfer from related languages, and then hurts every language, because each gets a smaller share of the capacity. Conneau et al. (2020) named this the curse of multilinguality. Their fix was scale: XLM-R Large has 550M parameters and was trained on 2.5 TB of filtered CommonCrawl text (CC-100). It beat mBERT by 14.6 points of average XNLI accuracy.

Models and benchmark

mBERT
BERT-base architecture; Wikipedia in 104 languages; shared WordPiece vocabulary of 110k; MLM + next-sentence prediction
XLM-R
Conneau et al. 2020; 100 languages, CC-100 (2.5 TB); SentencePiece 250k; Base 270M and Large 550M parameters; MLM only; α=0.3\alpha=0.3
mT5
Xue et al. 2021; T5 span corruption on mC4 in 101 languages; SentencePiece 250k; Small 300M to XXL 13B; α=0.3\alpha=0.3
XNLI
Conneau et al. 2018; natural language inference in 15 languages: 7,500 human-translated test and dev pairs per language; training data is English MultiNLI only
Settings
zero-shot (fine-tune in English, test everywhere), translate-train, translate-test; zero-shot is the cleanest test of cross-lingual representations

Worked example: α = 0.3 for English (300 GB) and Yoruba (0.1 GB) of 695.7 GB

  1. 1Natural shares: pen=300/695.7=0.4312p_\text{en}=300/695.7=0.4312, pyo=0.1/695.7=0.000144p_\text{yo}=0.1/695.7=0.000144.
  2. 2Exponentiate: 0.43120.3=0.77700.4312^{0.3}=0.7770 and 0.0001440.3=0.07040.000144^{0.3}=0.0704. The ratio between them falls from 30003000 to 11.011.0.
  3. 3Normalise: summing pj0.3p_j^{0.3} over all seven languages gives 2.98962.9896, so qen=0.7770/2.9896=0.2599q_\text{en}=0.7770/2.9896=0.2599 and qyo=0.0704/2.9896=0.0235q_\text{yo}=0.0704/2.9896=0.0235.
  4. 4Upsampling factors: Yoruba 0.0235/0.000144=163.7×0.0235/0.000144=163.7\times, English 0.2599/0.4312=0.60×0.2599/0.4312=0.60\times. Of 8,192 sampled sequences, about 193 are Yoruba (198 in the seeded draw below).
python
1import numpy as np
2
3sizes = {"en": 300, "ru": 280, "de": 66, "ar": 28, "hi": 20, "sw": 1.6, "yo": 0.1} # GB, illustrative
4n = np.array(list(sizes.values()))
5p = n / n.sum()
6
7def lang_probs(alpha):
8 w = p ** alpha
9 return w / w.sum()
10
11for alpha in (1.0, 0.7, 0.3):
12 q = lang_probs(alpha)
13 print(alpha, {k: round(float(v), 4) for k, v in zip(sizes, q)}, "upsampling yo:", round(q[-1] / p[-1], 1))
14# 1.0 {'en': 0.4312, 'ru': 0.4025, ..., 'yo': 0.0001} upsampling yo: 1.0
15# 0.7 {'en': 0.3747, 'ru': 0.357, ..., 'yo': 0.0014} upsampling yo: 9.6
16# 0.3 {'en': 0.2599, 'ru': 0.2546, ..., 'yo': 0.0235} upsampling yo: 163.7
17
18rng = np.random.default_rng(0)
19batch = rng.choice(list(sizes), size=8192, p=lang_probs(0.3)) # language of each sequence
20print({k: int((batch == k).sum()) for k in sizes})
21# {'en': 2163, 'ru': 2057, 'de': 1348, 'ar': 1030, 'hi': 974, 'sw': 422, 'yo': 198}
22
23# --- fertility of a real shared vocabulary (requires transformers) ---
24# from transformers import AutoTokenizer
25# tok = AutoTokenizer.from_pretrained("xlm-roberta-base")
26# for s in ["The cat sat on the mat.", "बिल्ली चटाई पर बैठी थी।"]:
27# pieces = tok.tokenize(s)
28# print(len(pieces) / len(s.split()), pieces)
Top (run; output in comments): exponent-smoothed language sampling. Bottom (not run here): measuring XLM-R tokenizer fertility

Use when

  • +One model serves 100+ languages, including ones with no labelled data
  • +Low-resource languages benefit from related high-resource ones
  • +No MT system is needed at training or test time for zero-shot transfer

Avoid when

  • −Curse of multilinguality: at a fixed size, per-language quality falls as languages are added
  • −High fertility for under-served scripts means longer sequences and higher cost per sentence
  • −A 250k vocabulary makes the embedding matrix a large share of the parameters (most of XLM-R Base)
  • −Usually behind a strong monolingual model of the same size in high-resource languages

Where it's used

Classification and NLI in many languages

Fine-tune once on English labels and deploy intent, topic or sentiment models in every pre-training language.

Multilingual NER and QA

Token tagging and extractive QA transferred zero-shot, or with a few hundred target-language examples to close most of the gap.

Encoders for metrics and retrieval

COMET builds on a multilingual encoder, and cross-lingual sentence embeddings retrieve documents across languages.

A small amount of target-language labelled data (few-shot) usually beats zero-shot by a wide margin. Zero-shot is the fallback for languages with none.

Check yourself

Two languages hold 300 GB and 0.1 GB of text. What is the ratio of their sampling probabilities at α = 0.3?

Check yourself

With a 512-token limit, roughly how many words fit for a language with fertility 3.0, versus one with fertility 1.3?

Remember

  • ✓One Transformer, one shared subword vocabulary, many languages, no language-specific parameters: mBERT (104 languages), XLM-R (100), mT5 (101).
  • ✓Language sampling qi∝piαq_i\propto p_i^{\alpha}: α=1\alpha=1 is natural, 00 uniform; mBERT 0.7, XLM 0.5, XLM-R and mT5 0.3.
  • ✓At α=0.3\alpha=0.3 a 0.1 GB language among 695.7 GB rises from 0.01% to 2.35% of training data, which means about 164 repeats.
  • ✓Zero-shot transfer: fine-tune on English (XNLI training data is English only) and test in 14 other languages. XLM-R beat mBERT by 14.6 points.
  • ✓Curse of multilinguality and tokenizer fertility: capacity and vocabulary are shared, so adding languages or under-covering a script costs quality.
← back to Deep Learning roadmap