# BERT

> BERT (Devlin et al., NAACL 2019) pre-trains a bidirectional transformer encoder by predicting 15% of WordPiece tokens (80% [MASK], 10% random, 10% unchanged) plus next-sentence prediction, then fine-tunes it with one output layer: base 12×768 (110M), large 24×1024 (340M), GLUE 80.5.

Source: https://www.mlfoundry.in/learn/bert  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**BERT:** (Bidirectional Encoder Representations from Transformers) is the pre-trained language representation of Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (NAACL 2019). It is a **Transformer encoder** stack with no causal mask, pre-trained on unlabeled text with two objectives. The **masked language model** (MLM) hides 15% of the input tokens and predicts them from both sides. **Next-sentence prediction** (NSP) classifies whether segment B really followed segment A. For each downstream task, one small output layer is added and every weight is fine-tuned.

Before BERT, transfer learning in NLP came in two forms. **Feature-based** methods such as ELMo (Peters et al., 2018) trained a left-to-right and a right-to-left LSTM language model separately and fed their concatenated states into a task-specific architecture. **Fine-tuning** methods such as OpenAI GPT (Radford et al., 2018) pre-trained a left-to-right transformer and fine-tuned all of it. Both were limited by the language-model objective. A left-to-right model predicts token $t$ from tokens $<t$ only, so its representation of a word never conditions on the words after it. ELMo's two directions are only joined shallowly, after each has been trained alone.

You cannot simply train a bidirectional encoder on next-token prediction: in a multi-layer bidirectional stack every token can attend to itself, so the target leaks and the task becomes trivial. BERT's answer is the cloze task (Taylor, 1953). Corrupt some input positions and ask the model to reconstruct them. Every layer can now attend in both directions, because the answers are not in the input. BERT-base was deliberately sized like GPT (12 layers, 768 wide, 12 heads) so that the comparison isolates the objective and the attention direction.

## Input representation

Every input is one or two **segments** of contiguous text packed into at most 512 WordPiece tokens. The first token is always `[CLS]`, and each segment ends with `[SEP]`. A segment is a span of text, not necessarily a linguistic sentence. The vocabulary has 30,000 WordPiece units (30,522 entries in the released uncased checkpoint); see **WordPiece**.

$$
h^{(0)}_i=\mathrm{LayerNorm}\big(E_{\text{tok}}[x_i]+E_{\text{seg}}[s_i]+E_{\text{pos}}[i]\big),\qquad s_i\in\{A,B\},\ i=0,\dots,511
$$

- $E_{\text{tok}}\in\mathbb R^{V\times H}$: WordPiece embedding, $V=30{,}522$; reused (tied) as the MLM output matrix
- $E_{\text{seg}}\in\mathbb R^{2\times H}$: segment embedding: $E_A$ for `[CLS]` through the first `[SEP]`, $E_B$ after it
- $E_{\text{pos}}\in\mathbb R^{512\times H}$: learned absolute position embedding (not sinusoidal)
- $h^{(L)}_0=C$: the final hidden state of `[CLS]`, used as the aggregate sequence representation

> Figure: BERT pre-training on the paper's own example pair. Each input position sums a token, a segment and a position embedding. A bidirectional encoder (12 or 24 post-LN layers) maps all positions at once. The [CLS] output C feeds the next-sentence classifier through a tanh pooler. The [MASK] position's output feeds the MLM head, whose output matrix is the token embedding table, and must recover "cute". Fine-tuning drops both heads and adds one task layer.

## Masked language model

The data generator picks 15% of the WordPiece positions in each sequence as prediction targets. Each picked position is then rewritten. With probability 0.8 it becomes `[MASK]`, with 0.1 a random token, and with 0.1 it is left unchanged. The loss is cross-entropy over the vocabulary at the picked positions only. The paper's example is `my dog is hairy` with `hairy` picked. It becomes `my dog is [MASK]` 80% of the time, `my dog is apple` 10% of the time, and stays `my dog is hairy` the other 10%.

$$
\mathcal L_{\text{MLM}}=-\frac{1}{|\mathcal M|}\sum_{i\in\mathcal M}\log\frac{\exp\big(e_{x_i}^{\top}g(h^{(L)}_i)+b_{x_i}\big)}{\sum_{w=1}^{V}\exp\big(e_w^{\top}g(h^{(L)}_i)+b_w\big)},\qquad |\mathcal M|\approx0.15\,T
$$

- $\mathcal M$: the picked positions; $x_i$ is the **original** token there, whatever the input shows
- $g(\cdot)$: the MLM transform: dense $H\to H$, GELU, LayerNorm
- $e_w$: row $w$ of $E_{\text{tok}}$ (tied output embedding); $b_w$ is a per-token output bias

Why not always use `[MASK]`? The symbol never appears at fine-tuning time, so a model that only learns to predict at `[MASK]` positions has no reason to build a good representation of a visible token. With the 80/10/10 rule, any visible token might be a target: with $\rho=0.15$, a non-`[MASK]` token is a hidden target with probability $0.15\cdot0.2/(1-0.12)=3.41\%$. The model therefore has to keep a contextual estimate of every token. Random replacement touches only $1.5\%$ of all tokens, which the authors judged too little to hurt language understanding. Their appendix ablation found the mixed strategy best. Always-`[MASK]` was noticeably worse for the feature-based NER setting, and always-random was worse than both.

The price is sample efficiency. A left-to-right model gets a training signal from every position, and BERT gets one from about 15%. The paper's Figure 5 shows that MLM does converge more slowly than a left-to-right model in pre-training steps, but it overtakes the left-to-right model in accuracy almost immediately.

## Next-sentence prediction

Question answering and natural language inference depend on the relation between two texts, which a language model does not learn directly. For each pre-training example, segment B is the actual next segment 50% of the time (`IsNext`) and a random segment from the corpus otherwise (`NotNext`). A classifier on $C$ predicts the label. The final model reaches 97–98% NSP accuracy. The total pre-training loss is the sum of the mean MLM likelihood and the mean NSP likelihood.

$$
\mathcal L_{\text{NSP}}=-\log\mathrm{softmax}\big(W_{\text{nsp}}\tanh(W_pC+b_p)\big)_{y},\qquad y\in\{\text{IsNext},\text{NotNext}\},\qquad \mathcal L=\mathcal L_{\text{MLM}}+\mathcal L_{\text{NSP}}
$$

- $\tanh(W_pC+b_p)$: the **pooler**, a dense $H\times H$ layer on `[CLS]`; it ships with the checkpoint
- $W_{\text{nsp}}\in\mathbb R^{2\times H}$: the binary NSP classifier, discarded after pre-training

NSP was later questioned. **RoBERTa** found that dropping it and packing full sentences from contiguous text matched or beat BERT, and **ALBERT** replaced it with sentence-order prediction, arguing that NSP is mostly topic detection. In BERT's own ablation on BERT-base, removing NSP lowered QNLI dev accuracy from 88.4 to 84.9, MNLI-m from 84.4 to 83.9 and SQuAD v1.1 F1 from 88.5 to 87.9. Replacing MLM with a left-to-right objective (also without NSP) cost far more: MRPC 86.7 → 77.5 and SQuAD F1 → 77.8.

**Configurations and training (Devlin et al. 2019)**

- **BERT-base:** $L=12$ layers, $H=768$, $A=12$ heads, FFN $4H=3072$, 110M parameters (109,482,240 counted with the pooler)
- **BERT-large:** $L=24$, $H=1024$, $A=16$, FFN 4096, 340M parameters (335,141,888 counted)
- **Block:** post-LN encoder block (as in Vaswani et al.), GELU activation, dropout 0.1 everywhere
- **Vocabulary:** WordPiece, 30,000 units (30,522 in the uncased release); max length 512
- **Data:** BooksCorpus (800M words) + English Wikipedia text passages (2,500M words); document-level, so long contiguous spans can be sampled
- **Pre-training:** 1,000,000 steps of 256 sequences × 512 tokens (128,000 tokens per batch), about 40 epochs over 3.3B words; 90% of steps at length 128, the last 10% at 512 to learn the positional embeddings
- **Optimiser:** Adam, lr $10^{-4}$, $\beta_1=0.9$, $\beta_2=0.999$, L2 weight decay 0.01, 10,000 warm-up steps then linear decay
- **Hardware:** base: 4 Cloud TPUs (16 chips); large: 16 Cloud TPUs (64 chips); 4 days each
- **Fine-tuning:** batch 16 or 32; lr $5\cdot10^{-5}$, $3\cdot10^{-5}$ or $2\cdot10^{-5}$; 2–4 epochs; at most an hour on one Cloud TPU for every task in the paper

> Figure: Top: the MLM corruption of the reference data generator applied to a 15-token pair. At 15% it picks round(15 · 0.15) = 2 non-special positions. Draw 0 masks "went" and "milk". Draw 2 keeps "he" unchanged as a target, and draw 3 turns "a" into the random piece "##ly" while keeping "gallon". The middle band is the exact expectation per 512 tokens: 76.8 targets = 61.4 [MASK] + 7.7 random + 7.7 kept, so 12% of inputs are [MASK] and a visible token is still a hidden target with p = 3.41% and wrong with p = 1.70%. At 100% [MASK] both probabilities fall to 0, so nothing forces the model to model visible tokens. At 0% they become 15% and 7.5%. Bottom: an exact parameter count. BERT-base has 109,482,240 parameters, with the token table at 23.44M (21.4%) and the encoder at 85.05M (77.7%; 7,087,872 per layer). BERT-large has 335,141,888, and its table falls to 9.3%. The rule of thumb 12·L·H² + V·H gives 108.38M for base.

## Fine-tuning: one extra layer per task

Fine-tuning changes the inputs and adds one output layer. Single sentences use segment A only. Sentence pairs (entailment, paraphrase) use A and B. Question answering puts the question in A and the passage in B. For classification, a weight matrix $W\in\mathbb R^{K\times H}$ is added and the loss is $\log\mathrm{softmax}(CW^{\top})$. Token-level tasks such as NER put a classifier on each $T_i$. For SQuAD the only new parameters are a start vector $S$ and an end vector $E$ in $\mathbb R^H$.

$$
P^{\text{start}}_i=\frac{e^{S\cdot T_i}}{\sum_j e^{S\cdot T_j}},\qquad \text{score}(i,j)=S\cdot T_i+E\cdot T_j\ \ (j\ge i),\qquad \hat s_{\text{null}}=S\cdot C+E\cdot C
$$

- $T_i$: final hidden state of passage token $i$
- $\text{score}(i,j)$: the predicted answer is the highest-scoring span with j ≥ i
- $\hat s_{\text{null}}$: SQuAD 2.0: "no answer" is the span at `[CLS]`; predict an answer only if the best span beats $\hat s_{\text{null}}$ by a threshold $\tau$ tuned on dev

## Worked example: masking budget and span extraction

**Masking one 128-token pre-training sequence**

1. The reference generator predicts $\mathrm{round}(128\cdot0.15)=19$ positions (capped at `max_predictions_per_seq` = 20 for length 128), never `[CLS]` or `[SEP]`.
2. Expected rewrites: $19\cdot0.8=15.2$ become `[MASK]`, $1.9$ become random tokens and $1.9$ stay unchanged. All 19 are in the loss.
3. The other 109 positions contribute nothing to $\mathcal L_{\text{MLM}}$ directly. Their only job is to be context.
4. Under the same 128 tokens, a left-to-right LM gets 127 targets. That is the 6.7× difference in supervision per sequence that MLM trades for bidirectionality.

Now fine-tuning on SQuAD. The question is "Where did the cat sit?" and the passage is `the cat sat on the mat today`. Suppose the fine-tuned model gives these start logits $S\cdot T_i$ and end logits $E\cdot T_j$.

Start and end scores for the passage tokens, with their softmax probabilities (computed in numpy)

| i | token | S·Tᵢ | P_start | E·Tᵢ | P_end |
| --- | --- | --- | --- | --- | --- |
| 0 | the | 0.1 | 0.073 | −0.6 | 0.030 |
| 1 | cat | 0.4 | 0.099 | 0.3 | 0.075 |
| 2 | sat | −0.2 | 0.054 | −0.1 | 0.050 |
| 3 | on | 2.1 | 0.542 | −0.3 | 0.041 |
| 4 | the | 0.9 | 0.163 | 0.2 | 0.068 |
| 5 | mat | −0.4 | 0.044 | 2.4 | 0.612 |
| 6 | today | −1.0 | 0.024 | 0.8 | 0.124 |

**Decoding and the training loss**

1. Best span: maximise $S\cdot T_i+E\cdot T_j$ over $j\ge i$. The winner is $i=3$, $j=5$ with $2.1+2.4=4.5$, i.e. **on the mat**. The runners-up are "the mat" (3.3), "on the mat today" (2.9) and "cat sat on the mat" (2.8).
2. Start softmax: $\log\sum_j e^{S\cdot T_j}=2.713$, so $P^{\text{start}}_3=e^{2.1-2.713}=0.542$. End softmax: $\log\sum_j e^{E\cdot T_j}=2.891$, so $P^{\text{end}}_5=e^{2.4-2.891}=0.612$.
3. If the gold span is (3, 5), the training loss is $-\log0.542-\log0.612=0.613+0.491=1.105$. The paper's objective is this sum of the correct start and end log-likelihoods.
4. The whole task head is $S$ and $E$: $2H=1{,}536$ new parameters on BERT-base, against 109.5M fine-tuned.

## Results

BERT-large set new results on eleven tasks. On the official GLUE leaderboard it scored 80.5, 7.7 points above the previous best. In the paper's Table 1 average (which excludes WNLI), BERT-base reached 79.6 and BERT-large 82.1, against 75.1 for OpenAI GPT. The base model has the same size as GPT, so most of that 4.5-point gap is the bidirectional objective. MultiNLI accuracy reached 86.7% (4.6 points absolute). SQuAD v1.1 test F1 reached 93.2 (an ensemble also trained on TriviaQA), and SQuAD v2.0 test F1 reached 83.1, 5.1 points above the previous system.

Scale helped even on small datasets. Going from 3 to 24 layers improved every GLUE task tried, including MRPC with 3,600 training examples. The features also worked without fine-tuning. On CoNLL-2003 NER, concatenating the top four hidden layers of BERT-base as fixed features gave 96.1 dev F1, against 96.4 with full fine-tuning.

```python
import numpy as np

CLS, SEP, MASK, PAD = 101, 102, 103, 0          # ids in bert-base-uncased's 30,522-token vocab
VOCAB = 30522

def mask_tokens(ids, rng, rate=0.15, p_mask=0.8, p_random=0.1):
    """BERT MLM corruption. ids: (B, T) int array. Returns inputs and labels (-100 = no loss)."""
    ids = np.asarray(ids)
    special = np.isin(ids, [CLS, SEP, PAD])
    chosen = (rng.random(ids.shape) < rate) & ~special          # 15% of real tokens become targets
    labels = np.where(chosen, ids, -100)

    inputs = ids.copy()
    u = rng.random(ids.shape)
    to_mask = chosen & (u < p_mask)                              # 80% -> [MASK]
    to_rand = chosen & (u >= p_mask) & (u < p_mask + p_random)   # 10% -> random token
    inputs[to_mask] = MASK                                       # remaining 10% stay unchanged
    inputs[to_rand] = rng.integers(1000, VOCAB, to_rand.sum())   # skip [unused]/special ids
    return inputs, labels

rng = np.random.default_rng(0)
body = rng.integers(1000, VOCAB, size=(4096, 126))               # 4,096 fake sequences of 128
ids = np.concatenate([np.full((4096, 1), CLS), body, np.full((4096, 1), SEP)], axis=1)
x, y = mask_tokens(ids, rng)

t = y != -100
print(f"targets  {t.mean():.4f} of positions, {t.sum(1).mean():.1f} per sequence")  # 0.1471, 18.8
print(f"[MASK]   {(x[t] == MASK).mean():.3f} of targets")                            # 0.798
print(f"random   {((x[t] != MASK) & (x[t] != ids[t])).mean():.3f}")                   # 0.103
print(f"kept     {(x[t] == ids[t]).mean():.3f}")                                      # 0.099
print(f"P(target | visible token) = {t[x != MASK].mean():.4f}")                       # 0.0336

# With Hugging Face the same corruption is DataCollatorForLanguageModeling(tok, mlm_probability=0.15),
# and a fine-tuning head is one line, e.g. BertForQuestionAnswering.from_pretrained("bert-base-uncased").
```

BERT's 80/10/10 MLM corruption in numpy, checked on 4,096 synthetic 128-token sequences (run; outputs shown)

> **Fine-tuning BERT-large is unstable on small data:** The paper reports that BERT-large fine-tuning was sometimes unstable on small datasets. The authors ran several random restarts and kept the best model on dev, with the same checkpoint but different data shuffling and classifier initialisation. If one run of a small GLUE task looks bad, try a few seeds and a smaller learning rate from the $\{5,3,2\}\cdot10^{-5}$ grid before blaming the method.

> **The 10% unchanged is what makes visible tokens matter:** If every target were `[MASK]`, the loss would never ask for the output at a visible position, and those outputs could ignore their own input. Fine-tuning sees only visible tokens. Keeping 10% of targets unchanged (and corrupting 10%) makes every visible token a possible target, with probability 3.41% at the default rates. The model must keep an accurate contextual estimate of each one, which is the representation fine-tuning uses.

> **BERT is not a text generator:** MLM predicts masked positions independently given the rest. There is no factorisation of $p(x)$ to sample from left to right, and every forward pass re-encodes the whole sequence with no **KV cache**. For generation, use a **Decoder-only models** design or an **Encoder–decoder models** design such as **T5** or **BART**. BERT-style **Encoder-only models** remain the cheap default for classification, tagging, retrieval embeddings and extractive QA.

**Strengths**

- Deep bidirectional context at every layer; GLUE 80.5 and SQuAD v1.1 F1 93.2 at release
- One pre-trained model fine-tunes to classification, tagging, pairs and span QA with a single extra layer
- Fine-tuning is cheap: 2–4 epochs, at most an hour on one Cloud TPU for each task in the paper
- Base model (110M) is small enough to serve at scale for classification and embeddings

**Limitations**

- Only about 15% of positions give a training signal, so pre-training is less sample-efficient than next-token LM
- [MASK] appears in pre-training but never at fine-tuning (pre-train/fine-tune mismatch)
- Masked tokens are predicted independently, so it cannot generate text or model joint dependencies between targets
- 512-token limit from learned positional embeddings; NSP turned out to be of little use (RoBERTa, ALBERT)

- **Sentence and pair classification:** Sentiment, NLI, paraphrase and intent: a softmax on $C$, fine-tuned end to end.
- **Token tagging:** NER, POS and slot filling with a classifier on every final hidden state.
- **Extractive QA:** SQuAD-style span prediction with start/end vectors $S$ and $E$.
- **Search and retrieval:** Cross-encoder re-ranking of query–passage pairs, and the backbone of many dense-embedding models.

**Question:** Why does BERT replace only 80% of the chosen tokens with [MASK], keeping 10% unchanged and swapping 10% for random tokens?

- A. To make the MLM loss converge faster than a left-to-right LM
- B. Because [MASK] never appears at fine-tuning time, so the model must also build good representations of visible tokens, any of which may be a target
- C. To reduce the vocabulary needed by the output softmax
- D. So that the NSP head can tell masked pairs from unmasked ones

**Answer:** B. Because [MASK] never appears at fine-tuning time, so the model must also build good representations of visible tokens, any of which may be a target — With always-`[MASK]`, outputs at visible positions are never supervised, and fine-tuning inputs contain no `[MASK]`. Mixing in unchanged and random targets means a visible token is a hidden target with probability $0.03/0.88=3.41\%$, so the model cannot ignore it.

**Question:** In BERT's own ablation on BERT-base, which change hurt SQuAD v1.1 dev F1 most (baseline 88.5)?

- A. Removing NSP (MLM only)
- B. Training left-to-right instead of MLM, without NSP
- C. Using 128-token sequences for 90% of the steps
- D. Tying the MLM output matrix to the token embeddings

**Answer:** B. Training left-to-right instead of MLM, without NSP — Removing NSP gave 87.9. Switching to a left-to-right objective dropped F1 to 77.8, because token representations no longer see the right context that span boundaries depend on. Adding a BiLSTM on top recovered part of it (84.9) but not all.

**Key points**

- BERT = a post-LN **Transformer encoder** with no causal mask: base $12\times768$, 12 heads, 110M; large $24\times1024$, 16 heads, 340M; 30K WordPiece, 512 positions.
- Input $=E_{\text{tok}}+E_{\text{seg}}+E_{\text{pos}}$ with `[CLS]` first and `[SEP]` after each segment; $C=h^{(L)}_{\text{[CLS]}}$ summarises the sequence.
- MLM picks 15% of tokens and rewrites them 80% `[MASK]`, 10% random, 10% unchanged. All picked positions are scored with a tied-softmax cross-entropy.
- NSP (50% real next segment, 50% random) is a binary classifier on C. Later work found it of little use.
- Fine-tuning adds one layer: $\mathrm{softmax}(CW^\top)$ for classification, per-token classifiers for tagging, and start/end vectors $S,E$ for span QA.
- Results: GLUE 80.5 (+7.7), MultiNLI 86.7%, SQuAD v1.1 test F1 93.2, SQuAD v2.0 test F1 83.1.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
