Definition
A multilingual model is a single network pre-trained on unlabelled text in many languages at once, with one shared subword vocabulary and no language-specific parameters. The main examples are mBERT (Devlin et al. 2018, 104 languages), XLM-R (Conneau et al. 2020, 100 languages) and mT5 (Xue et al. 2021, 101 languages). Its main use is zero-shot cross-lingual transfer: fine-tune on labelled data in one language, usually English, and apply the model to others.
Labelled data exists for a handful of languages, but users write in thousands. Before multilingual pre-training, the options were to translate the test input into English (translate-test), translate the English training set into each target language (translate-train), or align separately trained word embeddings with a bilingual dictionary. All of these need MT or dictionaries, and errors compound.
A multilingual model learns from monolingual text alone. mBERT and XLM-R use only masked language modelling on concatenated per-language corpora, with no parallel data and no language-ID input. Pires, Schlinger & Garrette (2019) still found that mBERT fine-tuned on one language works on others, even across scripts. Shared subwords (numbers, names, URLs), shared parameters, and the pressure to compress many languages into one space align their representations well enough to transfer.
Balancing languages: exponent sampling
Web text is dominated by a few languages. Sampling training batches in proportion to corpus size starves low-resource languages. Sampling uniformly over languages repeats a 0.1 GB corpus hundreds of times while seeing a fraction of English. Every major model interpolates between the two:
- amount of pre-training text in language i
- probability that a training sequence (or batch) is drawn from language i
- 1 keeps natural proportions, 0 is uniform; mBERT smooths with exponent 0.7, XLM (Lample & Conneau 2019) uses 0.5, XLM-R and mT5 use 0.3
The same smoothing is usually applied when training the shared vocabulary. Otherwise high-resource scripts take most of the pieces, and other languages are split into many short fragments.
A language's fertility is the average number of subword tokens per word. High fertility means longer sequences, more compute per sentence, less text per context window, and pieces that carry less meaning each. Rust et al. (2021) found that languages with poorly covered vocabularies did worse in multilingual models than in monolingual ones.
- a language; measured on the same (parallel) text across languages, fertility exposes how evenly the shared vocabulary serves them
The curse of multilinguality
At a fixed model size, adding languages helps low-resource ones at first, through transfer from related languages, and then hurts every language, because each gets a smaller share of the capacity. Conneau et al. (2020) named this the curse of multilinguality. Their fix was scale: XLM-R Large has 550M parameters and was trained on 2.5 TB of filtered CommonCrawl text (CC-100). It beat mBERT by 14.6 points of average XNLI accuracy.
Models and benchmark
- mBERT
- BERT-base architecture; Wikipedia in 104 languages; shared WordPiece vocabulary of 110k; MLM + next-sentence prediction
- XLM-R
- Conneau et al. 2020; 100 languages, CC-100 (2.5 TB); SentencePiece 250k; Base 270M and Large 550M parameters; MLM only;
- mT5
- Xue et al. 2021; T5 span corruption on mC4 in 101 languages; SentencePiece 250k; Small 300M to XXL 13B;
- XNLI
- Conneau et al. 2018; natural language inference in 15 languages: 7,500 human-translated test and dev pairs per language; training data is English MultiNLI only
- Settings
- zero-shot (fine-tune in English, test everywhere), translate-train, translate-test; zero-shot is the cleanest test of cross-lingual representations
Worked example: α = 0.3 for English (300 GB) and Yoruba (0.1 GB) of 695.7 GB
- 1Natural shares: , .
- 2Exponentiate: and . The ratio between them falls from to .
- 3Normalise: summing over all seven languages gives , so and .
- 4Upsampling factors: Yoruba , English . Of 8,192 sampled sequences, about 193 are Yoruba (198 in the seeded draw below).
1import numpy as np23sizes = {"en": 300, "ru": 280, "de": 66, "ar": 28, "hi": 20, "sw": 1.6, "yo": 0.1} # GB, illustrative4n = np.array(list(sizes.values()))5p = n / n.sum()67def lang_probs(alpha):8 w = p ** alpha9 return w / w.sum()1011for alpha in (1.0, 0.7, 0.3):12 q = lang_probs(alpha)13 print(alpha, {k: round(float(v), 4) for k, v in zip(sizes, q)}, "upsampling yo:", round(q[-1] / p[-1], 1))14# 1.0 {'en': 0.4312, 'ru': 0.4025, ..., 'yo': 0.0001} upsampling yo: 1.015# 0.7 {'en': 0.3747, 'ru': 0.357, ..., 'yo': 0.0014} upsampling yo: 9.616# 0.3 {'en': 0.2599, 'ru': 0.2546, ..., 'yo': 0.0235} upsampling yo: 163.71718rng = np.random.default_rng(0)19batch = rng.choice(list(sizes), size=8192, p=lang_probs(0.3)) # language of each sequence20print({k: int((batch == k).sum()) for k in sizes})21# {'en': 2163, 'ru': 2057, 'de': 1348, 'ar': 1030, 'hi': 974, 'sw': 422, 'yo': 198}2223# --- fertility of a real shared vocabulary (requires transformers) ---24# from transformers import AutoTokenizer25# tok = AutoTokenizer.from_pretrained("xlm-roberta-base")26# for s in ["The cat sat on the mat.", "बिल्ली चटाई पर बैठी थी।"]:27# pieces = tok.tokenize(s)28# print(len(pieces) / len(s.split()), pieces)Use when
- +One model serves 100+ languages, including ones with no labelled data
- +Low-resource languages benefit from related high-resource ones
- +No MT system is needed at training or test time for zero-shot transfer
Avoid when
- −Curse of multilinguality: at a fixed size, per-language quality falls as languages are added
- −High fertility for under-served scripts means longer sequences and higher cost per sentence
- −A 250k vocabulary makes the embedding matrix a large share of the parameters (most of XLM-R Base)
- −Usually behind a strong monolingual model of the same size in high-resource languages
Where it's used
Classification and NLI in many languages
Fine-tune once on English labels and deploy intent, topic or sentiment models in every pre-training language.
Multilingual NER and QA
Token tagging and extractive QA transferred zero-shot, or with a few hundred target-language examples to close most of the gap.
Encoders for metrics and retrieval
COMET builds on a multilingual encoder, and cross-lingual sentence embeddings retrieve documents across languages.
A small amount of target-language labelled data (few-shot) usually beats zero-shot by a wide margin. Zero-shot is the fallback for languages with none.
Check yourself
Two languages hold 300 GB and 0.1 GB of text. What is the ratio of their sampling probabilities at α = 0.3?
Check yourself
With a 512-token limit, roughly how many words fit for a language with fertility 3.0, versus one with fertility 1.3?
Remember
- ✓One Transformer, one shared subword vocabulary, many languages, no language-specific parameters: mBERT (104 languages), XLM-R (100), mT5 (101).
- ✓Language sampling : is natural, uniform; mBERT 0.7, XLM 0.5, XLM-R and mT5 0.3.
- ✓At a 0.1 GB language among 695.7 GB rises from 0.01% to 2.35% of training data, which means about 164 repeats.
- ✓Zero-shot transfer: fine-tune on English (XNLI training data is English only) and test in 14 other languages. XLM-R beat mBERT by 14.6 points.
- ✓Curse of multilinguality and tokenizer fertility: capacity and vocabulary are shared, so adding languages or under-covering a script costs quality.