RoBERTa

RoBERTa (Liu et al., 2019) is BERT's architecture with a better pre-training recipe: dynamic masking, no NSP, full 512-token inputs, 8K-sequence batches, 160GB of text, 50K byte-level BPE and 500K steps. RoBERTa-large reached GLUE 88.5, SQuAD 2.0 F1 89.8 and RACE 83.2.

14 min read1084 wordscore0 viewsupdated 2026-10-13

Definition

RoBERTa (Robustly optimized BERT approach) is the replication study of Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov, "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (arXiv 2019, Facebook AI and the University of Washington). It keeps BERT's architecture and masked-language-model objective unchanged. It retrains them with dynamic masking, no next-sentence prediction, full-length inputs, much larger batches, ten times more data, a byte-level BPE vocabulary and longer training. The paper's conclusion is that BERT was "significantly undertrained".

In 2019 several methods claimed to beat BERT: XLNet's permutation objective, SpanBERT's span masking and others. Each also changed the data, compute or hyperparameters. The authors asked a narrower question. How much of the gap disappears if you simply pre-train BERT properly? They reimplemented BERT in fairseq and varied one design choice at a time at BERT-base size. Then they combined the winners and scaled them up. The answer was most of it. RoBERTa-large matched or beat XLNet-large on GLUE, SQuAD and RACE with BERT's original objective, minus NSP.

So RoBERTa is not a new architecture. It is an argument about what counts as a fair comparison in pre-training: training data size, number of tokens processed and batch size all move downstream accuracy as much as many objective changes do. That lesson carried straight into the compute-optimal studies for decoder LLMs (see Pretraining and continued pretraining).

Same encoder (post-LN, L/H/A as BERT), different pre-training recipeBERT (Devlin et al. 2019)RoBERTa (Liu et al. 2019)maskingstatic: masked once offlinedynamic: new mask per feedobjectiveMLM + NSP on segment pairsMLM only, FULL-SENTENCESseq length128 for 90% of steps, then 512512 for every stepbatch × steps256 × 1M8K × 500K (large)data16GB: books + Wikipedia160GB: five corporavocabulary30K WordPiece50K byte-level BPEAdam β₂0.9990.98 (stable at large batch)sizebase 110M · large 340Mbase 125M · large 355MRoBERTa pre-training text: over 160GB of uncompressed English (BERT: books + Wikipedia only)CC-News76GBOpenWebText38GBStories31GBBooks+Wiki16GBCC-News: 63M English news articles crawled Sep 2016 – Feb 2019OpenWebText: pages linked from Reddit posts with ≥ 3 upvotes (open GPT-2 WebText recreation)Stories: CommonCrawl filtered to the story-like style of Winograd schemas · Books+Wiki: BERT's data
fig.Every change RoBERTa makes to BERT, row by row. The encoder itself is untouched (base: 12 layers, 768 wide, 12 heads; large: 24 layers, 1024 wide, 16 heads). The extra 15M (base) and 20M (large) parameters are all in the larger 50,265-entry embedding table. Below, the 160GB pre-training corpus. BERT's books + Wikipedia (16GB) is a tenth of it, and CC-News (76GB) is the largest part.

Change 1: dynamic masking

BERT's reference pipeline masks the data once, during preprocessing, so the same sequence comes back with the same mask every epoch. To soften this, the RoBERTa authors' BERT reimplementation duplicated the data 10 times with 10 different masks. Over 40 epochs, each sequence then appears with each of its masks 4 times. Dynamic masking draws a fresh mask every time a sequence is fed to the model, which is free when masking happens in the data loader.

Pr⁡[token i never a target after e epochs]=(1−ρ)min⁡(e, k),ρ=0.15,k={1one static mask10static, duplicated 10×∞dynamic\Pr[\text{token } i \text{ never a target after } e \text{ epochs}]=(1-\rho)^{\min(e,\,k)},\qquad \rho=0.15,\quad k=\begin{cases}1&\text{one static mask}\\10&\text{static, duplicated 10×}\\\infty&\text{dynamic}\end{cases}
ρ\rho
BERT's masking rate; the 80/10/10 rewrite rule is unchanged
kk
number of distinct masks a sequence ever gets
ee
epochs; BERT-base did about 40 over its corpus

With k=10k=10 and 40 epochs, 0.8510=19.7%0.85^{10}=19.7\% of tokens are never predicted at all, and every prediction is repeated four times. With dynamic masking only 0.8540=0.15%0.85^{40}=0.15\% are never predicted. The measured gain at BERT-base size was small: dynamic masking was "comparable or slightly better" than static (SQuAD 2.0 F1 78.7 vs 78.3, MNLI-m 84.0 vs 84.3, SST-2 92.9 vs 92.5). It matters more as training gets longer and data is reused, and it costs nothing, so every later run used it.

Change 2: drop next-sentence prediction

The paper tested four input formats at BERT-base size, all with about 512 tokens per input (Table 2, dev sets).

formatNSPSQuAD 1.1 / 2.0 F1MNLI-mSST-2RACE
SEGMENT-PAIR (BERT's format)yes90.4 / 78.784.092.964.2
SENTENCE-PAIR (single sentences)yes88.7 / 76.282.992.163.0
FULL-SENTENCES (may cross documents)no90.4 / 79.184.792.564.8
DOC-SENTENCES (one document only)no90.6 / 79.784.792.765.6
BERT-base as publishedyes88.5 / 76.384.392.864.3
Input format and NSP ablation (Liu et al. 2019, Table 2; BERT-base size, BOOKS + WIKI)

Two lessons. Pairs of single sentences hurt (SENTENCE-PAIR is worst), which the authors attribute to the model being unable to learn long-range dependencies. And removing the NSP loss matched or slightly improved downstream results, contradicting BERT's own ablation. That ablation had kept the SEGMENT-PAIR input format while dropping only the loss. DOC-SENTENCES was marginally best, but its batch size varies, so RoBERTa uses FULL-SENTENCES: contiguous sentences up to 512 tokens, with an extra separator at document boundaries.

Change 3: large batches, full length, longer training

BERT-base ran 1M steps of 256 sequences. At a fixed number of sequences seen, the paper compared three batch sizes at BERT-base size, retuning the learning rate for each (Table 3).

256×1M⏟lr 1e-4: ppl 3.99  =  2K×125K⏟lr 7e-4: ppl 3.68  ≈  8K×31K⏟lr 1e-3: ppl 3.77  ≈  2.56⋅108 sequences\underbrace{256\times1\text{M}}_{\text{lr }1\text{e-4: ppl }3.99}\;=\;\underbrace{2\text{K}\times125\text{K}}_{\text{lr }7\text{e-4: ppl }3.68}\;\approx\;\underbrace{8\text{K}\times31\text{K}}_{\text{lr }1\text{e-3: ppl }3.77}\;\approx\;2.56\cdot10^{8}\ \text{sequences}
ppl\text{ppl}
held-out MLM perplexity on the training-data distribution; lower is better
MNLI−m/SST−2MNLI-m / SST-2
84.7 / 92.7 at 256, 85.2 / 92.9 at 2K, 84.6 / 92.8 at 8K

Large batches gave equal or better models with 8–32× fewer sequential steps, and those steps are easy to spread across many GPUs with data parallelism (see Distributed LLM training). The authors set Adam's β2=0.98\beta_2=0.98 for stability at large batch sizes and trained on full 512-token sequences for every step. BERT used 128 tokens for 90% of steps. They then trained the large model far longer than BERT. Table 4 isolates each step: more data at the same 100K steps, then 300K steps, then 500K.

modeldatastepsSQuAD 1.1 / 2.0 F1MNLI-mSST-2
RoBERTa16GB100K93.6 / 87.389.095.3
+ more data160GB100K94.0 / 87.789.395.6
+ pretrain longer160GB300K94.4 / 88.790.096.1
+ pretrain even longer160GB500K94.6 / 89.490.296.4
BERT-large13GB1M × 25690.9 / 81.886.693.7
XLNet-large126GB500K × 2K94.5 / 88.889.895.6
RoBERTa-large as data and training length grow (Liu et al. 2019, Table 4; dev sets, batch 8K)

Even the first row, with BERT's data size, beats BERT-large by a wide margin, and that comes from the recipe alone. Ten times more data and five times more steps then add another 2.1 F1 on SQuAD 2.0 and 1.2 points on MNLI. Even the 500K-step model showed no sign of overfitting.

Change 4: byte-level BPE

BERT used a 30K WordPiece vocabulary built after heuristic pre-tokenisation. RoBERTa uses GPT-2's byte-level BPE with 50K units (50,265 entries in the released checkpoints). Any byte sequence can be encoded, so there is no [UNK] and no language-specific preprocessing. It adds (50,265−30,522)⋅H(50{,}265-30{,}522)\cdot H embedding parameters, about 15M for base and 20M for large. Early experiments showed it slightly worse on some tasks, but the authors chose it for its universality. See Byte-pair encoding (BPE).

Final RoBERTa configuration

Architecture
BERT's post-LN encoder. base: L=12L=12, H=768H=768, A=12A=12 (125M parameters); large: L=24L=24, H=1024H=1024, A=16A=16 (355M)
Data
over 160GB: BookCorpus + English Wikipedia 16GB, CC-News 76GB (63M news articles, Sep 2016 – Feb 2019), OpenWebText 38GB, Stories 31GB
Inputs
FULL-SENTENCES, up to 512 tokens every step, no NSP; dynamic 15% masking with the 80/10/10 rule
Vocabulary
byte-level BPE, 50K units, no pre-tokenisation rules
Optimiser
Adam β1=0.9\beta_1=0.9, β2=0.98\beta_2=0.98, ϵ=10−6\epsilon=10^{-6}, weight decay 0.01; peak lr 4⋅10−44\cdot10^{-4} (large) or 6⋅10−46\cdot10^{-4} (base) with 30K / 24K warm-up steps, then linear decay
Schedule
batch 8K sequences, 500K steps, mixed precision on DGX-1 nodes with 8 × 32GB V100 each; the 100K-step large run took about a day on 1024 V100s
Results (large)
GLUE leaderboard 88.5 (XLNet 88.4), best on 4 of 9 tasks; SQuAD 2.0 test F1 89.8 without extra QA data; RACE test accuracy 83.2
static, dup 10 · 40 epochs · ρ = 0.15 · one sequence of 32 tokensepoch ↓ / token position →140ever a target:0%50%100%110203040epochsshare of positions ever predicted, 1 − 0.85^min(e, k)after 40 epochs: expected coverage 80.3%, this sequence 23/32 = 71.9% (○)static, 1 mask: coverage 15.0% · distinct masks 1 · each mask seen 40×static, dup 10: coverage 80.3% · distinct masks 10 · each mask seen 4×dynamic: coverage 99.8% · distinct masks 40 · each mask seen 1×this sequence: 10 distinct patterns, 164 target predictions in totalpaper, BERT-base: static 78.3 / 84.3 / 92.5 · dynamic 78.7 / 84.0 / 92.9(SQuAD 2.0 F1 / MNLI-m / SST-2; reference BERT-base 76.3 / 84.3 / 92.8)
view
masking
fig.Masking view: one 32-token sequence over 40 epochs, one row per epoch, with picked positions coloured. With BERT-style static masking duplicated 10× (the default), rows repeat with period 10. After 40 epochs the expected share of positions ever predicted is 1 − 0.85¹⁰ = 80.3% (this sequence: 23/32 = 71.9%), with 10 distinct masks each seen 4×. A single static mask stays at 15.0% (seen 40×). Dynamic masking reaches 1 − 0.85⁴⁰ ≈ 99.8% (this sequence: all 32), with 40 distinct masks. All three agree for the first epoch, and static-10 and dynamic agree up to epoch 10 (80.3%). Batch view: a fixed budget of 256M sequences (131.1B tokens of 512). Batch 256 needs 1M steps, 2K needs 125,000 and 8K needs 31,250. The paper's measured perplexities are 3.99, 3.68 and 3.77, with tuned learning rates 1e-4, 7e-4 and 1e-3. Linear scaling from 1e-4 would predict 8e-4 at 2K and 3.2e-3 at 8K.

Worked example: what BERT and RoBERTa actually processed

Token budgets

  1. 1BERT-base as published: 90% of 1M steps at length 128 and 10% at 512, with batch 256. That is 0.9⋅106⋅256⋅128=29.50.9\cdot10^6\cdot256\cdot128=29.5B plus 0.1⋅106⋅256⋅512=13.10.1\cdot10^6\cdot256\cdot512=13.1B, so about 42.6B tokens.
  2. 2RoBERTa's BERT-base baseline (full 512-token inputs): 256⋅512⋅106=131.1256\cdot512\cdot10^6=131.1B tokens, three times BERT's own count.
  3. 3Final RoBERTa: 8192⋅512⋅500,000=2.108192\cdot512\cdot500{,}000=2.10T tokens. That is 16× the sequences of the 256 × 1M baseline and about 49× BERT-base's real token count.
  4. 4Per step: BERT moves 256⋅512=131,072256\cdot512=131{,}072 tokens per update and RoBERTa moves 4,194,3044{,}194{,}304, 32× more, with half as many updates (500K vs 1M).
epochs estatic, 1 maskstatic, dup 10dynamic
115.0%15.0%15.0%
215.0%27.8%27.8%
415.0%47.8%47.8%
1015.0%80.3%80.3%
2015.0%80.3%96.1%
4015.0%80.3%99.85%
Share of a sequence's tokens that have been an MLM target at least once, ρ = 0.15 (1 − 0.85^min(e, k), computed in numpy)
python
1import numpy as np
2
3BOS, EOS, MASK, VOCAB = 0, 2, 50264, 50265 # roberta-base ids: <s>, </s>, <mask>, vocab size
4T = 512
5
6def full_sentences(docs, T=T):
7 """RoBERTa FULL-SENTENCES packing: fill each 512-token input with contiguous sentences,
8 crossing document boundaries (marked with </s>), and no segment-pair / NSP label."""
9 stream = []
10 for doc in docs:
11 for sent in doc:
12 stream.extend(sent)
13 stream.append(EOS) # extra separator between documents
14 n = len(stream) // (T - 2)
15 body = np.array(stream[: n * (T - 2)]).reshape(n, T - 2)
16 return np.concatenate([np.full((n, 1), BOS), body, np.full((n, 1), EOS)], axis=1)
17
18def dynamic_mask(batch, rng, rate=0.15):
19 """Drawn at collate time, so every epoch sees a new mask (80/10/10 as in BERT)."""
20 special = (batch == BOS) | (batch == EOS)
21 picked = (rng.random(batch.shape) < rate) & ~special
22 u = rng.random(batch.shape)
23 x = batch.copy()
24 x[picked & (u < 0.8)] = MASK
25 rand = picked & (u >= 0.8) & (u < 0.9)
26 x[rand] = rng.integers(3, VOCAB - 1, rand.sum())
27 return x, np.where(picked, batch, -100)
28
29rng = np.random.default_rng(0)
30docs = [[list(rng.integers(3, 50000, rng.integers(8, 40))) for _ in range(rng.integers(3, 30))] for _ in range(2000)]
31data = full_sentences(docs)
32print(data.shape) # (1456, 512)
33
34for name, dup in [("static (1 mask)", 1), ("static (dup 10)", 10), ("dynamic", None)]:
35 masks = [dynamic_mask(data, np.random.default_rng(k))[1] != -100 for k in range(10 if dup else 40)]
36 ever = np.zeros(data.shape, bool)
37 for epoch in range(40):
38 ever |= masks[epoch % dup] if dup else masks[epoch]
39 real = (data != BOS) & (data != EOS)
40 print(f"{name:16s} share of tokens ever predicted after 40 epochs: {ever[real].mean():.4f}")
41# static (1 mask) ... 0.1499
42# static (dup 10) ... 0.8029
43# dynamic ... 0.9985
44
45# In Hugging Face, DataCollatorForLanguageModeling(RobertaTokenizerFast.from_pretrained("roberta-base"))
46# masks inside the collator, so it is dynamic masking by construction.
FULL-SENTENCES packing and dynamic masking in numpy on 2,000 synthetic documents, and the coverage of static vs dynamic masks after 40 epochs (run; outputs shown)

Use when

  • +Same architecture and inference cost as BERT, with a much stronger checkpoint (MNLI-m 90.2 vs 86.6 for BERT-large)
  • +Simpler objective: MLM only, no NSP sampling of segment pairs
  • +Byte-level BPE needs no [UNK] and no language-specific pre-tokenisation
  • +Large batches cut sequential steps 8–32× and parallelise well

Avoid when

  • −Pre-training cost far above BERT's: about 49× BERT-base's real token count
  • −Bigger embedding table (+15M base, +20M large) and still a 512-token limit
  • −Still an encoder: no generation, and ~15% of positions supervised per pass
  • −Large-batch training needs retuned learning rates and Adam β₂

Check yourself

Under static masking with the data duplicated 10 times and trained for 40 epochs, roughly what share of tokens is never a prediction target (ρ = 0.15)?

Check yourself

Which finding made RoBERTa remove next-sentence prediction?

Remember

  • ✓RoBERTa = BERT's architecture and MLM, retrained: dynamic masking, no NSP, FULL-SENTENCES inputs of 512, batch 8K, 160GB of text, 50K byte-level BPE, 500K steps.
  • ✓Static masking with kk masks leaves 0.85k0.85^{k} of tokens never predicted (k=10k=10: 19.7%); dynamic masking leaves 0.85e0.85^{e} after ee epochs.
  • ✓At equal sequences seen, batch 2K × 125K steps beat 256 × 1M (perplexity 3.68 vs 3.99) with 8× fewer sequential steps.
  • ✓More data (16 → 160GB) and longer training (100K → 500K steps) each kept improving results, with no sign of overfitting.
  • ✓Results (large): GLUE 88.5, SQuAD 2.0 test F1 89.8, RACE 83.2, matching XLNet without changing the objective. BERT was undertrained.
← back to Deep Learning roadmap