# Multilingual models

> One Transformer pre-trained on 100+ languages with one shared subword vocabulary and p^α language sampling (mBERT, XLM-R, mT5): fine-tune it in English and it transfers zero-shot to languages it never saw labels for.

Source: https://www.mlfoundry.in/learn/multilingual-models  
Section: Deep Learning › Deep Learning for NLP  
Level: advanced · 8 min read · updated 2026-10-15

**A multilingual model:** is a single network pre-trained on unlabelled text in many languages at once, with one shared subword vocabulary and no language-specific parameters. The main examples are **mBERT** (Devlin et al. 2018, 104 languages), **XLM-R** (Conneau et al. 2020, 100 languages) and **mT5** (Xue et al. 2021, 101 languages). Its main use is **zero-shot cross-lingual transfer**: fine-tune on labelled data in one language, usually English, and apply the model to others.

Labelled data exists for a handful of languages, but users write in thousands. Before multilingual pre-training, the options were to translate the test input into English (*translate-test*), translate the English training set into each target language (*translate-train*), or align separately trained word embeddings with a bilingual dictionary. All of these need MT or dictionaries, and errors compound.

A multilingual model learns from monolingual text alone. mBERT and XLM-R use only **masked language modelling** on concatenated per-language corpora, with no parallel data and no language-ID input. Pires, Schlinger & Garrette (2019) still found that mBERT fine-tuned on one language works on others, even across scripts. Shared subwords (numbers, names, URLs), shared parameters, and the pressure to compress many languages into one space align their representations well enough to transfer.

> Figure: The pipeline. Per-language corpora are very unbalanced (illustrative sizes: English 300 GB, Yoruba 0.1 GB) and are resampled with q ∝ p^α, which at α = 0.3 lifts Yoruba from 0.01% to 2.4% of training data. One SentencePiece vocabulary of 250k pieces covers every script. One Transformer is pre-trained with masked language modelling, fine-tuned on English XNLI only, and tested on the 14 other XNLI languages. Below: the tokenizer's raw input differs by script. The same short sentence is 23 bytes in English, 44 in Russian and 61 in Hindi.

## Balancing languages: exponent sampling

Web text is dominated by a few languages. Sampling training batches in proportion to corpus size starves low-resource languages. Sampling uniformly over languages repeats a 0.1 GB corpus hundreds of times while seeing a fraction of English. Every major model interpolates between the two:

$$
q_i=\frac{p_i^{\alpha}}{\sum_{j}p_j^{\alpha}},\qquad p_i=\frac{n_i}{\sum_j n_j}
$$

- $n_i$: amount of pre-training text in language i
- $q_i$: probability that a training sequence (or batch) is drawn from language i
- $\alpha$: 1 keeps natural proportions, 0 is uniform; mBERT smooths with exponent 0.7, XLM (Lample & Conneau 2019) uses 0.5, XLM-R and mT5 use 0.3

The same smoothing is usually applied when training the shared vocabulary. Otherwise high-resource scripts take most of the pieces, and other languages are split into many short fragments.

A language's **fertility** is the average number of subword tokens per word. High fertility means longer sequences, more compute per sentence, less text per context window, and pieces that carry less meaning each. Rust et al. (2021) found that languages with poorly covered vocabularies did worse in multilingual models than in monolingual ones.

$$
\text{fertility}(\ell)=\frac{\#\,\text{subword tokens of a text in }\ell}{\#\,\text{words of that text}}
$$

- $\ell$: a language; measured on the same (parallel) text across languages, fertility exposes how evenly the shared vocabulary serves them

> Figure: Seven languages with illustrative corpus sizes (695.7 GB in total). At the default α = 0.3 (XLM-R, mT5), English drops from 43.1% of the data to 26.0% and Yoruba rises from 0.01% to 2.35%. If training reads as much text as the corpus holds, Yoruba is seen 163.7 times and English 0.60 times, and a 256-sequence batch holds 6.02 Yoruba sequences instead of 0.04. The mixture is equivalent to 5.78 evenly used languages (exp of its entropy). The mBERT preset (0.7) gives Yoruba only 0.14% (×9.6). Uniform (α = 0) gives every language 14.29% and repeats Yoruba 993.9 times.

## The curse of multilinguality

At a fixed model size, adding languages helps low-resource ones at first, through transfer from related languages, and then hurts every language, because each gets a smaller share of the capacity. Conneau et al. (2020) named this the **curse of multilinguality**. Their fix was scale: XLM-R Large has 550M parameters and was trained on 2.5 TB of filtered CommonCrawl text (CC-100). It beat mBERT by 14.6 points of average XNLI accuracy.

**Models and benchmark**

- **mBERT:** BERT-base architecture; Wikipedia in 104 languages; shared WordPiece vocabulary of 110k; MLM + next-sentence prediction
- **XLM-R:** Conneau et al. 2020; 100 languages, CC-100 (2.5 TB); SentencePiece 250k; Base 270M and Large 550M parameters; MLM only; $\alpha=0.3$
- **mT5:** Xue et al. 2021; T5 span corruption on mC4 in 101 languages; SentencePiece 250k; Small 300M to XXL 13B; $\alpha=0.3$
- **XNLI:** Conneau et al. 2018; natural language inference in 15 languages: 7,500 human-translated test and dev pairs per language; training data is English MultiNLI only
- **Settings:** zero-shot (fine-tune in English, test everywhere), translate-train, translate-test; zero-shot is the cleanest test of cross-lingual representations

**Worked example: α = 0.3 for English (300 GB) and Yoruba (0.1 GB) of 695.7 GB**

1. Natural shares: $p_\text{en}=300/695.7=0.4312$, $p_\text{yo}=0.1/695.7=0.000144$.
2. Exponentiate: $0.4312^{0.3}=0.7770$ and $0.000144^{0.3}=0.0704$. The ratio between them falls from $3000$ to $11.0$.
3. Normalise: summing $p_j^{0.3}$ over all seven languages gives $2.9896$, so $q_\text{en}=0.7770/2.9896=0.2599$ and $q_\text{yo}=0.0704/2.9896=0.0235$.
4. Upsampling factors: Yoruba $0.0235/0.000144=163.7\times$, English $0.2599/0.4312=0.60\times$. Of 8,192 sampled sequences, about 193 are Yoruba (198 in the seeded draw below).

```python
import numpy as np

sizes = {"en": 300, "ru": 280, "de": 66, "ar": 28, "hi": 20, "sw": 1.6, "yo": 0.1}  # GB, illustrative
n = np.array(list(sizes.values()))
p = n / n.sum()

def lang_probs(alpha):
    w = p ** alpha
    return w / w.sum()

for alpha in (1.0, 0.7, 0.3):
    q = lang_probs(alpha)
    print(alpha, {k: round(float(v), 4) for k, v in zip(sizes, q)}, "upsampling yo:", round(q[-1] / p[-1], 1))
# 1.0 {'en': 0.4312, 'ru': 0.4025, ..., 'yo': 0.0001} upsampling yo: 1.0
# 0.7 {'en': 0.3747, 'ru': 0.357,  ..., 'yo': 0.0014} upsampling yo: 9.6
# 0.3 {'en': 0.2599, 'ru': 0.2546, ..., 'yo': 0.0235} upsampling yo: 163.7

rng = np.random.default_rng(0)
batch = rng.choice(list(sizes), size=8192, p=lang_probs(0.3))   # language of each sequence
print({k: int((batch == k).sum()) for k in sizes})
# {'en': 2163, 'ru': 2057, 'de': 1348, 'ar': 1030, 'hi': 974, 'sw': 422, 'yo': 198}

# --- fertility of a real shared vocabulary (requires transformers) ---
# from transformers import AutoTokenizer
# tok = AutoTokenizer.from_pretrained("xlm-roberta-base")
# for s in ["The cat sat on the mat.", "बिल्ली चटाई पर बैठी थी।"]:
#     pieces = tok.tokenize(s)
#     print(len(pieces) / len(s.split()), pieces)
```

Top (run; output in comments): exponent-smoothed language sampling. Bottom (not run here): measuring XLM-R tokenizer fertility

> **Low α is not free:** Pushing $\alpha$ toward 0 makes a 0.1 GB language appear hundreds of times per pass, and the model memorises it instead of generalising. It also cuts the share of high-resource languages, which lowers their scores. Values of 0.3–0.7 are a compromise, not an optimum for every language.

> **Why transfer works without parallel data:** Nothing in MLM tells the model that *Katze* and *cat* mean the same. But identical anchor tokens, shared parameters, and limited capacity make reusing one representation per concept cheaper than learning 100 separate ones. Fine-tuning a classifier on English representations then works, imperfectly, on aligned representations in other languages. Accuracy still falls with linguistic distance from English and with smaller pre-training data.

**Strengths**

- One model serves 100+ languages, including ones with no labelled data
- Low-resource languages benefit from related high-resource ones
- No MT system is needed at training or test time for zero-shot transfer

**Limitations**

- Curse of multilinguality: at a fixed size, per-language quality falls as languages are added
- High fertility for under-served scripts means longer sequences and higher cost per sentence
- A 250k vocabulary makes the embedding matrix a large share of the parameters (most of XLM-R Base)
- Usually behind a strong monolingual model of the same size in high-resource languages

- **Classification and NLI in many languages:** Fine-tune once on English labels and deploy intent, topic or sentiment models in every pre-training language.
- **Multilingual NER and QA:** Token tagging and extractive QA transferred zero-shot, or with a few hundred target-language examples to close most of the gap.
- **Encoders for metrics and retrieval:** COMET builds on a multilingual encoder, and cross-lingual sentence embeddings retrieve documents across languages.

A small amount of target-language labelled data (few-shot) usually beats zero-shot by a wide margin. Zero-shot is the fallback for languages with none.

**Question:** Two languages hold 300 GB and 0.1 GB of text. What is the ratio of their sampling probabilities at α = 0.3?

- A. 3000
- B. (3000)^0.3 ≈ 11.0
- C. 3000 × 0.3 = 900
- D. 1, because α < 0.5 is uniform

**Answer:** B. (3000)^0.3 ≈ 11.0 — $q_1/q_2=(p_1/p_2)^{\alpha}=3000^{0.3}=11.0$. The normaliser cancels in the ratio. That matches the worked example: $0.7770/0.0704$.

**Question:** With a 512-token limit, roughly how many words fit for a language with fertility 3.0, versus one with fertility 1.3?

- A. 512 vs 512
- B. about 171 vs about 394
- C. about 1,536 vs about 666
- D. about 394 vs about 171

**Answer:** B. about 171 vs about 394 — Words that fit = 512 / fertility: 512/3.0 ≈ 171 and 512/1.3 ≈ 394. The high-fertility language gets less than half the context and pays more compute per word.

**Key points**

- One Transformer, one shared subword vocabulary, many languages, no language-specific parameters: mBERT (104 languages), XLM-R (100), mT5 (101).
- Language sampling $q_i\propto p_i^{\alpha}$: $\alpha=1$ is natural, $0$ uniform; mBERT 0.7, XLM 0.5, XLM-R and mT5 0.3.
- At $\alpha=0.3$ a 0.1 GB language among 695.7 GB rises from 0.01% to 2.35% of training data, which means about 164 repeats.
- Zero-shot transfer: fine-tune on English (XNLI training data is English only) and test in 14 other languages. XLM-R beat mBERT by 14.6 points.
- Curse of multilinguality and tokenizer fertility: capacity and vocabulary are shared, so adding languages or under-covering a script costs quality.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
