Definition
RoBERTa (Robustly optimized BERT approach) is the replication study of Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov, "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (arXiv 2019, Facebook AI and the University of Washington). It keeps BERT's architecture and masked-language-model objective unchanged. It retrains them with dynamic masking, no next-sentence prediction, full-length inputs, much larger batches, ten times more data, a byte-level BPE vocabulary and longer training. The paper's conclusion is that BERT was "significantly undertrained".
In 2019 several methods claimed to beat BERT: XLNet's permutation objective, SpanBERT's span masking and others. Each also changed the data, compute or hyperparameters. The authors asked a narrower question. How much of the gap disappears if you simply pre-train BERT properly? They reimplemented BERT in fairseq and varied one design choice at a time at BERT-base size. Then they combined the winners and scaled them up. The answer was most of it. RoBERTa-large matched or beat XLNet-large on GLUE, SQuAD and RACE with BERT's original objective, minus NSP.
So RoBERTa is not a new architecture. It is an argument about what counts as a fair comparison in pre-training: training data size, number of tokens processed and batch size all move downstream accuracy as much as many objective changes do. That lesson carried straight into the compute-optimal studies for decoder LLMs (see Pretraining and continued pretraining).
Change 1: dynamic masking
BERT's reference pipeline masks the data once, during preprocessing, so the same sequence comes back with the same mask every epoch. To soften this, the RoBERTa authors' BERT reimplementation duplicated the data 10 times with 10 different masks. Over 40 epochs, each sequence then appears with each of its masks 4 times. Dynamic masking draws a fresh mask every time a sequence is fed to the model, which is free when masking happens in the data loader.
- BERT's masking rate; the 80/10/10 rewrite rule is unchanged
- number of distinct masks a sequence ever gets
- epochs; BERT-base did about 40 over its corpus
With and 40 epochs, of tokens are never predicted at all, and every prediction is repeated four times. With dynamic masking only are never predicted. The measured gain at BERT-base size was small: dynamic masking was "comparable or slightly better" than static (SQuAD 2.0 F1 78.7 vs 78.3, MNLI-m 84.0 vs 84.3, SST-2 92.9 vs 92.5). It matters more as training gets longer and data is reused, and it costs nothing, so every later run used it.
Change 2: drop next-sentence prediction
The paper tested four input formats at BERT-base size, all with about 512 tokens per input (Table 2, dev sets).
| format | NSP | SQuAD 1.1 / 2.0 F1 | MNLI-m | SST-2 | RACE |
|---|---|---|---|---|---|
| SEGMENT-PAIR (BERT's format) | yes | 90.4 / 78.7 | 84.0 | 92.9 | 64.2 |
| SENTENCE-PAIR (single sentences) | yes | 88.7 / 76.2 | 82.9 | 92.1 | 63.0 |
| FULL-SENTENCES (may cross documents) | no | 90.4 / 79.1 | 84.7 | 92.5 | 64.8 |
| DOC-SENTENCES (one document only) | no | 90.6 / 79.7 | 84.7 | 92.7 | 65.6 |
| BERT-base as published | yes | 88.5 / 76.3 | 84.3 | 92.8 | 64.3 |
Two lessons. Pairs of single sentences hurt (SENTENCE-PAIR is worst), which the authors attribute to the model being unable to learn long-range dependencies. And removing the NSP loss matched or slightly improved downstream results, contradicting BERT's own ablation. That ablation had kept the SEGMENT-PAIR input format while dropping only the loss. DOC-SENTENCES was marginally best, but its batch size varies, so RoBERTa uses FULL-SENTENCES: contiguous sentences up to 512 tokens, with an extra separator at document boundaries.
Change 3: large batches, full length, longer training
BERT-base ran 1M steps of 256 sequences. At a fixed number of sequences seen, the paper compared three batch sizes at BERT-base size, retuning the learning rate for each (Table 3).
- held-out MLM perplexity on the training-data distribution; lower is better
- 84.7 / 92.7 at 256, 85.2 / 92.9 at 2K, 84.6 / 92.8 at 8K
Large batches gave equal or better models with 8–32× fewer sequential steps, and those steps are easy to spread across many GPUs with data parallelism (see Distributed LLM training). The authors set Adam's for stability at large batch sizes and trained on full 512-token sequences for every step. BERT used 128 tokens for 90% of steps. They then trained the large model far longer than BERT. Table 4 isolates each step: more data at the same 100K steps, then 300K steps, then 500K.
| model | data | steps | SQuAD 1.1 / 2.0 F1 | MNLI-m | SST-2 |
|---|---|---|---|---|---|
| RoBERTa | 16GB | 100K | 93.6 / 87.3 | 89.0 | 95.3 |
| + more data | 160GB | 100K | 94.0 / 87.7 | 89.3 | 95.6 |
| + pretrain longer | 160GB | 300K | 94.4 / 88.7 | 90.0 | 96.1 |
| + pretrain even longer | 160GB | 500K | 94.6 / 89.4 | 90.2 | 96.4 |
| BERT-large | 13GB | 1M × 256 | 90.9 / 81.8 | 86.6 | 93.7 |
| XLNet-large | 126GB | 500K × 2K | 94.5 / 88.8 | 89.8 | 95.6 |
Even the first row, with BERT's data size, beats BERT-large by a wide margin, and that comes from the recipe alone. Ten times more data and five times more steps then add another 2.1 F1 on SQuAD 2.0 and 1.2 points on MNLI. Even the 500K-step model showed no sign of overfitting.
Change 4: byte-level BPE
BERT used a 30K WordPiece vocabulary built after heuristic pre-tokenisation. RoBERTa uses GPT-2's byte-level BPE with 50K units (50,265 entries in the released checkpoints). Any byte sequence can be encoded, so there is no [UNK] and no language-specific preprocessing. It adds embedding parameters, about 15M for base and 20M for large. Early experiments showed it slightly worse on some tasks, but the authors chose it for its universality. See Byte-pair encoding (BPE).
Final RoBERTa configuration
- Architecture
- BERT's post-LN encoder. base: , , (125M parameters); large: , , (355M)
- Data
- over 160GB: BookCorpus + English Wikipedia 16GB, CC-News 76GB (63M news articles, Sep 2016 – Feb 2019), OpenWebText 38GB, Stories 31GB
- Inputs
- FULL-SENTENCES, up to 512 tokens every step, no NSP; dynamic 15% masking with the 80/10/10 rule
- Vocabulary
- byte-level BPE, 50K units, no pre-tokenisation rules
- Optimiser
- Adam , , , weight decay 0.01; peak lr (large) or (base) with 30K / 24K warm-up steps, then linear decay
- Schedule
- batch 8K sequences, 500K steps, mixed precision on DGX-1 nodes with 8 × 32GB V100 each; the 100K-step large run took about a day on 1024 V100s
- Results (large)
- GLUE leaderboard 88.5 (XLNet 88.4), best on 4 of 9 tasks; SQuAD 2.0 test F1 89.8 without extra QA data; RACE test accuracy 83.2
Worked example: what BERT and RoBERTa actually processed
Token budgets
- 1BERT-base as published: 90% of 1M steps at length 128 and 10% at 512, with batch 256. That is B plus B, so about 42.6B tokens.
- 2RoBERTa's BERT-base baseline (full 512-token inputs): B tokens, three times BERT's own count.
- 3Final RoBERTa: T tokens. That is 16× the sequences of the 256 × 1M baseline and about 49× BERT-base's real token count.
- 4Per step: BERT moves tokens per update and RoBERTa moves , 32× more, with half as many updates (500K vs 1M).
| epochs e | static, 1 mask | static, dup 10 | dynamic |
|---|---|---|---|
| 1 | 15.0% | 15.0% | 15.0% |
| 2 | 15.0% | 27.8% | 27.8% |
| 4 | 15.0% | 47.8% | 47.8% |
| 10 | 15.0% | 80.3% | 80.3% |
| 20 | 15.0% | 80.3% | 96.1% |
| 40 | 15.0% | 80.3% | 99.85% |
1import numpy as np23BOS, EOS, MASK, VOCAB = 0, 2, 50264, 50265 # roberta-base ids: <s>, </s>, <mask>, vocab size4T = 51256def full_sentences(docs, T=T):7 """RoBERTa FULL-SENTENCES packing: fill each 512-token input with contiguous sentences,8 crossing document boundaries (marked with </s>), and no segment-pair / NSP label."""9 stream = []10 for doc in docs:11 for sent in doc:12 stream.extend(sent)13 stream.append(EOS) # extra separator between documents14 n = len(stream) // (T - 2)15 body = np.array(stream[: n * (T - 2)]).reshape(n, T - 2)16 return np.concatenate([np.full((n, 1), BOS), body, np.full((n, 1), EOS)], axis=1)1718def dynamic_mask(batch, rng, rate=0.15):19 """Drawn at collate time, so every epoch sees a new mask (80/10/10 as in BERT)."""20 special = (batch == BOS) | (batch == EOS)21 picked = (rng.random(batch.shape) < rate) & ~special22 u = rng.random(batch.shape)23 x = batch.copy()24 x[picked & (u < 0.8)] = MASK25 rand = picked & (u >= 0.8) & (u < 0.9)26 x[rand] = rng.integers(3, VOCAB - 1, rand.sum())27 return x, np.where(picked, batch, -100)2829rng = np.random.default_rng(0)30docs = [[list(rng.integers(3, 50000, rng.integers(8, 40))) for _ in range(rng.integers(3, 30))] for _ in range(2000)]31data = full_sentences(docs)32print(data.shape) # (1456, 512)3334for name, dup in [("static (1 mask)", 1), ("static (dup 10)", 10), ("dynamic", None)]:35 masks = [dynamic_mask(data, np.random.default_rng(k))[1] != -100 for k in range(10 if dup else 40)]36 ever = np.zeros(data.shape, bool)37 for epoch in range(40):38 ever |= masks[epoch % dup] if dup else masks[epoch]39 real = (data != BOS) & (data != EOS)40 print(f"{name:16s} share of tokens ever predicted after 40 epochs: {ever[real].mean():.4f}")41# static (1 mask) ... 0.149942# static (dup 10) ... 0.802943# dynamic ... 0.99854445# In Hugging Face, DataCollatorForLanguageModeling(RobertaTokenizerFast.from_pretrained("roberta-base"))46# masks inside the collator, so it is dynamic masking by construction.Use when
- +Same architecture and inference cost as BERT, with a much stronger checkpoint (MNLI-m 90.2 vs 86.6 for BERT-large)
- +Simpler objective: MLM only, no NSP sampling of segment pairs
- +Byte-level BPE needs no [UNK] and no language-specific pre-tokenisation
- +Large batches cut sequential steps 8–32× and parallelise well
Avoid when
- −Pre-training cost far above BERT's: about 49× BERT-base's real token count
- −Bigger embedding table (+15M base, +20M large) and still a 512-token limit
- −Still an encoder: no generation, and ~15% of positions supervised per pass
- −Large-batch training needs retuned learning rates and Adam β₂
Check yourself
Under static masking with the data duplicated 10 times and trained for 40 epochs, roughly what share of tokens is never a prediction target (ρ = 0.15)?
Check yourself
Which finding made RoBERTa remove next-sentence prediction?
Remember
- ✓RoBERTa = BERT's architecture and MLM, retrained: dynamic masking, no NSP, FULL-SENTENCES inputs of 512, batch 8K, 160GB of text, 50K byte-level BPE, 500K steps.
- ✓Static masking with masks leaves of tokens never predicted (: 19.7%); dynamic masking leaves after epochs.
- ✓At equal sequences seen, batch 2K × 125K steps beat 256 × 1M (perplexity 3.68 vs 3.99) with 8× fewer sequential steps.
- ✓More data (16 → 160GB) and longer training (100K → 500K steps) each kept improving results, with no sign of overfitting.
- ✓Results (large): GLUE 88.5, SQuAD 2.0 test F1 89.8, RACE 83.2, matching XLNet without changing the objective. BERT was undertrained.