# RoBERTa

> RoBERTa (Liu et al., 2019) is BERT's architecture with a better pre-training recipe: dynamic masking, no NSP, full 512-token inputs, 8K-sequence batches, 160GB of text, 50K byte-level BPE and 500K steps. RoBERTa-large reached GLUE 88.5, SQuAD 2.0 F1 89.8 and RACE 83.2.

Source: https://www.mlfoundry.in/learn/roberta  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**RoBERTa:** (Robustly optimized BERT approach) is the replication study of Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov, "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (arXiv 2019, Facebook AI and the University of Washington). It keeps **BERT**'s architecture and masked-language-model objective unchanged. It retrains them with **dynamic masking**, **no next-sentence prediction**, full-length inputs, **much larger batches**, **ten times more data**, a byte-level BPE vocabulary and **longer training**. The paper's conclusion is that BERT was "significantly undertrained".

In 2019 several methods claimed to beat BERT: XLNet's permutation objective, SpanBERT's span masking and others. Each also changed the data, compute or hyperparameters. The authors asked a narrower question. How much of the gap disappears if you simply pre-train BERT properly? They reimplemented BERT in fairseq and varied one design choice at a time at BERT-base size. Then they combined the winners and scaled them up. The answer was most of it. RoBERTa-large matched or beat XLNet-large on GLUE, SQuAD and RACE with BERT's original objective, minus NSP.

So RoBERTa is not a new architecture. It is an argument about **what counts as a fair comparison** in pre-training: training data size, number of tokens processed and batch size all move downstream accuracy as much as many objective changes do. That lesson carried straight into the compute-optimal studies for decoder LLMs (see **Pretraining and continued pretraining**).

> Figure: Every change RoBERTa makes to BERT, row by row. The encoder itself is untouched (base: 12 layers, 768 wide, 12 heads; large: 24 layers, 1024 wide, 16 heads). The extra 15M (base) and 20M (large) parameters are all in the larger 50,265-entry embedding table. Below, the 160GB pre-training corpus. BERT's books + Wikipedia (16GB) is a tenth of it, and CC-News (76GB) is the largest part.

## Change 1: dynamic masking

BERT's reference pipeline masks the data once, during preprocessing, so the same sequence comes back with the **same mask** every epoch. To soften this, the RoBERTa authors' BERT reimplementation duplicated the data 10 times with 10 different masks. Over 40 epochs, each sequence then appears with each of its masks 4 times. **Dynamic masking** draws a fresh mask every time a sequence is fed to the model, which is free when masking happens in the data loader.

$$
\Pr[\text{token } i \text{ never a target after } e \text{ epochs}]=(1-\rho)^{\min(e,\,k)},\qquad \rho=0.15,\quad k=\begin{cases}1&\text{one static mask}\\10&\text{static, duplicated 10×}\\\infty&\text{dynamic}\end{cases}
$$

- $\rho$: BERT's masking rate; the 80/10/10 rewrite rule is unchanged
- $k$: number of distinct masks a sequence ever gets
- $e$: epochs; BERT-base did about 40 over its corpus

With $k=10$ and 40 epochs, $0.85^{10}=19.7\%$ of tokens are never predicted at all, and every prediction is repeated four times. With dynamic masking only $0.85^{40}=0.15\%$ are never predicted. The measured gain at BERT-base size was small: dynamic masking was "comparable or slightly better" than static (SQuAD 2.0 F1 78.7 vs 78.3, MNLI-m 84.0 vs 84.3, SST-2 92.9 vs 92.5). It matters more as training gets longer and data is reused, and it costs nothing, so every later run used it.

## Change 2: drop next-sentence prediction

The paper tested four input formats at BERT-base size, all with about 512 tokens per input (Table 2, dev sets).

Input format and NSP ablation (Liu et al. 2019, Table 2; BERT-base size, BOOKS + WIKI)

| format | NSP | SQuAD 1.1 / 2.0 F1 | MNLI-m | SST-2 | RACE |
| --- | --- | --- | --- | --- | --- |
| SEGMENT-PAIR (BERT's format) | yes | 90.4 / 78.7 | 84.0 | 92.9 | 64.2 |
| SENTENCE-PAIR (single sentences) | yes | 88.7 / 76.2 | 82.9 | 92.1 | 63.0 |
| FULL-SENTENCES (may cross documents) | no | 90.4 / 79.1 | 84.7 | 92.5 | 64.8 |
| DOC-SENTENCES (one document only) | no | 90.6 / 79.7 | 84.7 | 92.7 | 65.6 |
| BERT-base as published | yes | 88.5 / 76.3 | 84.3 | 92.8 | 64.3 |

Two lessons. Pairs of single sentences hurt (SENTENCE-PAIR is worst), which the authors attribute to the model being unable to learn long-range dependencies. And removing the NSP loss matched or slightly improved downstream results, contradicting BERT's own ablation. That ablation had kept the SEGMENT-PAIR input format while dropping only the loss. DOC-SENTENCES was marginally best, but its batch size varies, so RoBERTa uses **FULL-SENTENCES**: contiguous sentences up to 512 tokens, with an extra separator at document boundaries.

## Change 3: large batches, full length, longer training

BERT-base ran 1M steps of 256 sequences. At a fixed number of sequences seen, the paper compared three batch sizes at BERT-base size, retuning the learning rate for each (Table 3).

$$
\underbrace{256\times1\text{M}}_{\text{lr }1\text{e-4: ppl }3.99}\;=\;\underbrace{2\text{K}\times125\text{K}}_{\text{lr }7\text{e-4: ppl }3.68}\;\approx\;\underbrace{8\text{K}\times31\text{K}}_{\text{lr }1\text{e-3: ppl }3.77}\;\approx\;2.56\cdot10^{8}\ \text{sequences}
$$

- $\text{ppl}$: held-out MLM perplexity on the training-data distribution; lower is better
- $MNLI-m / SST-2$: 84.7 / 92.7 at 256, 85.2 / 92.9 at 2K, 84.6 / 92.8 at 8K

Large batches gave equal or better models with 8–32× fewer sequential steps, and those steps are easy to spread across many GPUs with data parallelism (see **Distributed LLM training**). The authors set Adam's $\beta_2=0.98$ for stability at large batch sizes and trained on full 512-token sequences for every step. BERT used 128 tokens for 90% of steps. They then trained the large model far longer than BERT. Table 4 isolates each step: more data at the same 100K steps, then 300K steps, then 500K.

RoBERTa-large as data and training length grow (Liu et al. 2019, Table 4; dev sets, batch 8K)

| model | data | steps | SQuAD 1.1 / 2.0 F1 | MNLI-m | SST-2 |
| --- | --- | --- | --- | --- | --- |
| RoBERTa | 16GB | 100K | 93.6 / 87.3 | 89.0 | 95.3 |
| + more data | 160GB | 100K | 94.0 / 87.7 | 89.3 | 95.6 |
| + pretrain longer | 160GB | 300K | 94.4 / 88.7 | 90.0 | 96.1 |
| + pretrain even longer | 160GB | 500K | 94.6 / 89.4 | 90.2 | 96.4 |
| BERT-large | 13GB | 1M × 256 | 90.9 / 81.8 | 86.6 | 93.7 |
| XLNet-large | 126GB | 500K × 2K | 94.5 / 88.8 | 89.8 | 95.6 |

Even the first row, with BERT's data size, beats BERT-large by a wide margin, and that comes from the recipe alone. Ten times more data and five times more steps then add another 2.1 F1 on SQuAD 2.0 and 1.2 points on MNLI. Even the 500K-step model showed no sign of overfitting.

## Change 4: byte-level BPE

BERT used a 30K WordPiece vocabulary built after heuristic pre-tokenisation. RoBERTa uses GPT-2's **byte-level BPE** with 50K units (50,265 entries in the released checkpoints). Any byte sequence can be encoded, so there is no `[UNK]` and no language-specific preprocessing. It adds $(50{,}265-30{,}522)\cdot H$ embedding parameters, about 15M for base and 20M for large. Early experiments showed it slightly worse on some tasks, but the authors chose it for its universality. See **Byte-pair encoding (BPE)**.

**Final RoBERTa configuration**

- **Architecture:** BERT's post-LN encoder. base: $L=12$, $H=768$, $A=12$ (125M parameters); large: $L=24$, $H=1024$, $A=16$ (355M)
- **Data:** over 160GB: BookCorpus + English Wikipedia 16GB, CC-News 76GB (63M news articles, Sep 2016 – Feb 2019), OpenWebText 38GB, Stories 31GB
- **Inputs:** FULL-SENTENCES, up to 512 tokens every step, no NSP; dynamic 15% masking with the 80/10/10 rule
- **Vocabulary:** byte-level BPE, 50K units, no pre-tokenisation rules
- **Optimiser:** Adam $\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-6}$, weight decay 0.01; peak lr $4\cdot10^{-4}$ (large) or $6\cdot10^{-4}$ (base) with 30K / 24K warm-up steps, then linear decay
- **Schedule:** batch 8K sequences, 500K steps, mixed precision on DGX-1 nodes with 8 × 32GB V100 each; the 100K-step large run took about a day on 1024 V100s
- **Results (large):** GLUE leaderboard 88.5 (XLNet 88.4), best on 4 of 9 tasks; SQuAD 2.0 test F1 89.8 without extra QA data; RACE test accuracy 83.2

> Figure: Masking view: one 32-token sequence over 40 epochs, one row per epoch, with picked positions coloured. With BERT-style static masking duplicated 10× (the default), rows repeat with period 10. After 40 epochs the expected share of positions ever predicted is 1 − 0.85¹⁰ = 80.3% (this sequence: 23/32 = 71.9%), with 10 distinct masks each seen 4×. A single static mask stays at 15.0% (seen 40×). Dynamic masking reaches 1 − 0.85⁴⁰ ≈ 99.8% (this sequence: all 32), with 40 distinct masks. All three agree for the first epoch, and static-10 and dynamic agree up to epoch 10 (80.3%). Batch view: a fixed budget of 256M sequences (131.1B tokens of 512). Batch 256 needs 1M steps, 2K needs 125,000 and 8K needs 31,250. The paper's measured perplexities are 3.99, 3.68 and 3.77, with tuned learning rates 1e-4, 7e-4 and 1e-3. Linear scaling from 1e-4 would predict 8e-4 at 2K and 3.2e-3 at 8K.

## Worked example: what BERT and RoBERTa actually processed

**Token budgets**

1. BERT-base as published: 90% of 1M steps at length 128 and 10% at 512, with batch 256. That is $0.9\cdot10^6\cdot256\cdot128=29.5$B plus $0.1\cdot10^6\cdot256\cdot512=13.1$B, so about **42.6B tokens**.
2. RoBERTa's BERT-base baseline (full 512-token inputs): $256\cdot512\cdot10^6=131.1$B tokens, three times BERT's own count.
3. Final RoBERTa: $8192\cdot512\cdot500{,}000=2.10$T tokens. That is 16× the sequences of the 256 × 1M baseline and about 49× BERT-base's real token count.
4. Per step: BERT moves $256\cdot512=131{,}072$ tokens per update and RoBERTa moves $4{,}194{,}304$, 32× more, with half as many updates (500K vs 1M).

Share of a sequence's tokens that have been an MLM target at least once, ρ = 0.15 (1 − 0.85^min(e, k), computed in numpy)

| epochs e | static, 1 mask | static, dup 10 | dynamic |
| --- | --- | --- | --- |
| 1 | 15.0% | 15.0% | 15.0% |
| 2 | 15.0% | 27.8% | 27.8% |
| 4 | 15.0% | 47.8% | 47.8% |
| 10 | 15.0% | 80.3% | 80.3% |
| 20 | 15.0% | 80.3% | 96.1% |
| 40 | 15.0% | 80.3% | 99.85% |

```python
import numpy as np

BOS, EOS, MASK, VOCAB = 0, 2, 50264, 50265       # roberta-base ids: <s>, </s>, <mask>, vocab size
T = 512

def full_sentences(docs, T=T):
    """RoBERTa FULL-SENTENCES packing: fill each 512-token input with contiguous sentences,
    crossing document boundaries (marked with </s>), and no segment-pair / NSP label."""
    stream = []
    for doc in docs:
        for sent in doc:
            stream.extend(sent)
        stream.append(EOS)                          # extra separator between documents
    n = len(stream) // (T - 2)
    body = np.array(stream[: n * (T - 2)]).reshape(n, T - 2)
    return np.concatenate([np.full((n, 1), BOS), body, np.full((n, 1), EOS)], axis=1)

def dynamic_mask(batch, rng, rate=0.15):
    """Drawn at collate time, so every epoch sees a new mask (80/10/10 as in BERT)."""
    special = (batch == BOS) | (batch == EOS)
    picked = (rng.random(batch.shape) < rate) & ~special
    u = rng.random(batch.shape)
    x = batch.copy()
    x[picked & (u < 0.8)] = MASK
    rand = picked & (u >= 0.8) & (u < 0.9)
    x[rand] = rng.integers(3, VOCAB - 1, rand.sum())
    return x, np.where(picked, batch, -100)

rng = np.random.default_rng(0)
docs = [[list(rng.integers(3, 50000, rng.integers(8, 40))) for _ in range(rng.integers(3, 30))] for _ in range(2000)]
data = full_sentences(docs)
print(data.shape)                                   # (1456, 512)

for name, dup in [("static (1 mask)", 1), ("static (dup 10)", 10), ("dynamic", None)]:
    masks = [dynamic_mask(data, np.random.default_rng(k))[1] != -100 for k in range(10 if dup else 40)]
    ever = np.zeros(data.shape, bool)
    for epoch in range(40):
        ever |= masks[epoch % dup] if dup else masks[epoch]
    real = (data != BOS) & (data != EOS)
    print(f"{name:16s} share of tokens ever predicted after 40 epochs: {ever[real].mean():.4f}")
# static (1 mask)  ... 0.1499
# static (dup 10)  ... 0.8029
# dynamic          ... 0.9985

# In Hugging Face, DataCollatorForLanguageModeling(RobertaTokenizerFast.from_pretrained("roberta-base"))
# masks inside the collator, so it is dynamic masking by construction.
```

FULL-SENTENCES packing and dynamic masking in numpy on 2,000 synthetic documents, and the coverage of static vs dynamic masks after 40 epochs (run; outputs shown)

> **Compare pre-training methods at equal data and compute:** RoBERTa's first row (16GB, 100K steps of 8K) already beats published BERT-large on SQuAD 2.0 by 5.5 F1, using BERT's own objective. Many 2019 "BERT killers" were partly comparisons between a new objective and an undertrained baseline. When you read an ablation, check the tokens processed, the batch size, the data size and the learning-rate tuning before crediting the objective.

> **Why NSP looked useful in BERT:** BERT's "no NSP" ablation dropped the loss but kept the input format: two shorter segments, each possibly half a document. RoBERTa found that the format was what hurt. Single-sentence pairs lose up to 2.5 points against segment pairs (SQuAD 2.0 F1 76.2 vs 78.7). With FULL-SENTENCES inputs, removing NSP is neutral or helpful. The objective seemed to matter because it was confounded with input length.

> **Same checkpoints, different tokenizer:** RoBERTa uses `<s>` and `</s>` instead of `[CLS]` and `[SEP]`, has no segment (token-type) embeddings in practice, and its byte-level BPE marks a leading space as part of the token (`Ġthe`). Code written for BERT's WordPiece ids or `token_type_ids` needs changes. Always load the tokenizer that matches the checkpoint.

**Strengths**

- Same architecture and inference cost as BERT, with a much stronger checkpoint (MNLI-m 90.2 vs 86.6 for BERT-large)
- Simpler objective: MLM only, no NSP sampling of segment pairs
- Byte-level BPE needs no [UNK] and no language-specific pre-tokenisation
- Large batches cut sequential steps 8–32× and parallelise well

**Limitations**

- Pre-training cost far above BERT's: about 49× BERT-base's real token count
- Bigger embedding table (+15M base, +20M large) and still a 512-token limit
- Still an encoder: no generation, and ~15% of positions supervised per pass
- Large-batch training needs retuned learning rates and Adam β₂

**Question:** Under static masking with the data duplicated 10 times and trained for 40 epochs, roughly what share of tokens is never a prediction target (ρ = 0.15)?

- A. 0.15%
- B. about 20%
- C. about 50%
- D. 85%

**Answer:** B. about 20% — Each sequence has only 10 distinct masks, so a token escapes all of them with probability $0.85^{10}=0.197$, about 20%. Dynamic masking draws 40 masks, giving $0.85^{40}=0.15\%$. A single static mask leaves 85% never predicted.

**Question:** Which finding made RoBERTa remove next-sentence prediction?

- A. NSP accuracy was too low to give a useful gradient
- B. With full 512-token inputs from contiguous text, dropping the NSP loss matched or improved downstream results; single-sentence pairs were what hurt
- C. NSP is incompatible with dynamic masking
- D. Byte-level BPE cannot encode the [SEP] token

**Answer:** B. With full 512-token inputs from contiguous text, dropping the NSP loss matched or improved downstream results; single-sentence pairs were what hurt — In Table 2, FULL-SENTENCES and DOC-SENTENCES without NSP matched or beat SEGMENT-PAIR with NSP (e.g. MNLI-m 84.7 vs 84.0), while SENTENCE-PAIR with NSP was worst (82.9). BERT's earlier ablation had changed the loss without changing the input format.

**Key points**

- RoBERTa = BERT's architecture and MLM, retrained: dynamic masking, no NSP, FULL-SENTENCES inputs of 512, batch 8K, 160GB of text, 50K byte-level BPE, 500K steps.
- Static masking with $k$ masks leaves $0.85^{k}$ of tokens never predicted ($k=10$: 19.7%); dynamic masking leaves $0.85^{e}$ after $e$ epochs.
- At equal sequences seen, batch 2K × 125K steps beat 256 × 1M (perplexity 3.68 vs 3.99) with 8× fewer sequential steps.
- More data (16 → 160GB) and longer training (100K → 500K steps) each kept improving results, with no sign of overfitting.
- Results (large): GLUE 88.5, SQuAD 2.0 test F1 89.8, RACE 83.2, matching XLNet without changing the objective. BERT was undertrained.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
