BERT

BERT (Devlin et al., NAACL 2019) pre-trains a bidirectional transformer encoder by predicting 15% of WordPiece tokens (80% [MASK], 10% random, 10% unchanged) plus next-sentence prediction, then fine-tunes it with one output layer: base 12×768 (110M), large 24×1024 (340M), GLUE 80.5.

15 min read1495 wordscore0 viewsupdated 2026-10-13

Definition

BERT (Bidirectional Encoder Representations from Transformers) is the pre-trained language representation of Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (NAACL 2019). It is a Transformer encoder stack with no causal mask, pre-trained on unlabeled text with two objectives. The masked language model (MLM) hides 15% of the input tokens and predicts them from both sides. Next-sentence prediction (NSP) classifies whether segment B really followed segment A. For each downstream task, one small output layer is added and every weight is fine-tuned.

Before BERT, transfer learning in NLP came in two forms. Feature-based methods such as ELMo (Peters et al., 2018) trained a left-to-right and a right-to-left LSTM language model separately and fed their concatenated states into a task-specific architecture. Fine-tuning methods such as OpenAI GPT (Radford et al., 2018) pre-trained a left-to-right transformer and fine-tuned all of it. Both were limited by the language-model objective. A left-to-right model predicts token tt from tokens <t<t only, so its representation of a word never conditions on the words after it. ELMo's two directions are only joined shallowly, after each has been trained alone.

You cannot simply train a bidirectional encoder on next-token prediction: in a multi-layer bidirectional stack every token can attend to itself, so the target leaks and the task becomes trivial. BERT's answer is the cloze task (Taylor, 1953). Corrupt some input positions and ask the model to reconstruct them. Every layer can now attend in both directions, because the answers are not in the input. BERT-base was deliberately sized like GPT (12 layers, 768 wide, 12 heads) so that the comparison isolates the objective and the attention direction.

Input representation

Every input is one or two segments of contiguous text packed into at most 512 WordPiece tokens. The first token is always [CLS], and each segment ends with [SEP]. A segment is a span of text, not necessarily a linguistic sentence. The vocabulary has 30,000 WordPiece units (30,522 entries in the released uncased checkpoint); see WordPiece.

hi(0)=LayerNorm(Etok[xi]+Eseg[si]+Epos[i]),si∈{A,B}, i=0,…,511h^{(0)}_i=\mathrm{LayerNorm}\big(E_{\text{tok}}[x_i]+E_{\text{seg}}[s_i]+E_{\text{pos}}[i]\big),\qquad s_i\in\{A,B\},\ i=0,\dots,511
Etok∈RV×HE_{\text{tok}}\in\mathbb R^{V\times H}
WordPiece embedding, V=30,522V=30{,}522; reused (tied) as the MLM output matrix
Eseg∈R2×HE_{\text{seg}}\in\mathbb R^{2\times H}
segment embedding: EAE_A for [CLS] through the first [SEP], EBE_B after it
Epos∈R512×HE_{\text{pos}}\in\mathbb R^{512\times H}
learned absolute position embedding (not sinusoidal)
h0(L)=Ch^{(L)}_0=C
the final hidden state of [CLS], used as the aggregate sequence representation
token V×H30,522 WordPiecesegment 2×HE_A / E_Bposition 512×Hlearned[CLS]mydogis[MASK][SEP]helikesplay##ing[SEP]E_AE_AE_AE_AE_AE_AE_BE_BE_BE_BE_BE_0E_1E_2E_3E_4E_5E_6E_7E_8E_9E_10++++++++++++++++++++++sum → LayerNorm→ dropout 0.1Transformer encoder × L · bidirectional self-attentionpost-LN blocks: MHA → add & norm → FFN (4H, GELU) → add & normBASE: L=12, H=768, A=12 · 110MLARGE: L=24, H=1024, A=16 · 340Mevery tokensees everyother tokenCT1T2T3T4T5T6T7T8T9T10final hiddenstates ∈ ℝ^HNSP: pooler tanh(W·C)→ IsNext / NotNextMLM: GELU dense + LN → Eᵀ (tied)softmax over 30,522 → “cute”loss = MLM cross-entropy onthe 15% chosen + NSP CEpre-training heads (dropped for fine-tuning, which adds one task layer)
fig.BERT pre-training on the paper's own example pair. Each input position sums a token, a segment and a position embedding. A bidirectional encoder (12 or 24 post-LN layers) maps all positions at once. The [CLS] output C feeds the next-sentence classifier through a tanh pooler. The [MASK] position's output feeds the MLM head, whose output matrix is the token embedding table, and must recover "cute". Fine-tuning drops both heads and adds one task layer.

Masked language model

The data generator picks 15% of the WordPiece positions in each sequence as prediction targets. Each picked position is then rewritten. With probability 0.8 it becomes [MASK], with 0.1 a random token, and with 0.1 it is left unchanged. The loss is cross-entropy over the vocabulary at the picked positions only. The paper's example is my dog is hairy with hairy picked. It becomes my dog is [MASK] 80% of the time, my dog is apple 10% of the time, and stays my dog is hairy the other 10%.

LMLM=−1∣M∣∑i∈Mlog⁡exp⁡(exi⊤g(hi(L))+bxi)∑w=1Vexp⁡(ew⊤g(hi(L))+bw),∣M∣≈0.15 T\mathcal L_{\text{MLM}}=-\frac{1}{|\mathcal M|}\sum_{i\in\mathcal M}\log\frac{\exp\big(e_{x_i}^{\top}g(h^{(L)}_i)+b_{x_i}\big)}{\sum_{w=1}^{V}\exp\big(e_w^{\top}g(h^{(L)}_i)+b_w\big)},\qquad |\mathcal M|\approx0.15\,T
M\mathcal M
the picked positions; xix_i is the original token there, whatever the input shows
g(⋅)g(\cdot)
the MLM transform: dense H→HH\to H, GELU, LayerNorm
ewe_w
row ww of EtokE_{\text{tok}} (tied output embedding); bwb_w is a per-token output bias

Why not always use [MASK]? The symbol never appears at fine-tuning time, so a model that only learns to predict at [MASK] positions has no reason to build a good representation of a visible token. With the 80/10/10 rule, any visible token might be a target: with ρ=0.15\rho=0.15, a non-[MASK] token is a hidden target with probability 0.15⋅0.2/(1−0.12)=3.41%0.15\cdot0.2/(1-0.12)=3.41\%. The model therefore has to keep a contextual estimate of every token. Random replacement touches only 1.5%1.5\% of all tokens, which the authors judged too little to hurt language understanding. Their appendix ablation found the mixed strategy best. Always-[MASK] was noticeably worse for the feature-based NER setting, and always-random was worse than both.

The price is sample efficiency. A left-to-right model gets a training signal from every position, and BERT gets one from about 15%. The paper's Figure 5 shows that MLM does converge more slowly than a left-to-right model in pre-training steps, but it overtakes the left-to-right model in accuracy almost immediately.

Next-sentence prediction

Question answering and natural language inference depend on the relation between two texts, which a language model does not learn directly. For each pre-training example, segment B is the actual next segment 50% of the time (IsNext) and a random segment from the corpus otherwise (NotNext). A classifier on CC predicts the label. The final model reaches 97–98% NSP accuracy. The total pre-training loss is the sum of the mean MLM likelihood and the mean NSP likelihood.

LNSP=−log⁡softmax(Wnsptanh⁡(WpC+bp))y,y∈{IsNext,NotNext},L=LMLM+LNSP\mathcal L_{\text{NSP}}=-\log\mathrm{softmax}\big(W_{\text{nsp}}\tanh(W_pC+b_p)\big)_{y},\qquad y\in\{\text{IsNext},\text{NotNext}\},\qquad \mathcal L=\mathcal L_{\text{MLM}}+\mathcal L_{\text{NSP}}
tanh⁡(WpC+bp)\tanh(W_pC+b_p)
the pooler, a dense H×HH\times H layer on [CLS]; it ships with the checkpoint
Wnsp∈R2×HW_{\text{nsp}}\in\mathbb R^{2\times H}
the binary NSP classifier, discarded after pre-training

NSP was later questioned. RoBERTa found that dropping it and packing full sentences from contiguous text matched or beat BERT, and ALBERT replaced it with sentence-order prediction, arguing that NSP is mostly topic detection. In BERT's own ablation on BERT-base, removing NSP lowered QNLI dev accuracy from 88.4 to 84.9, MNLI-m from 84.4 to 83.9 and SQuAD v1.1 F1 from 88.5 to 87.9. Replacing MLM with a left-to-right objective (also without NSP) cost far more: MRPC 86.7 → 77.5 and SQuAD F1 → 77.8.

Configurations and training (Devlin et al. 2019)

BERT-base
L=12L=12 layers, H=768H=768, A=12A=12 heads, FFN 4H=30724H=3072, 110M parameters (109,482,240 counted with the pooler)
BERT-large
L=24L=24, H=1024H=1024, A=16A=16, FFN 4096, 340M parameters (335,141,888 counted)
Block
post-LN encoder block (as in Vaswani et al.), GELU activation, dropout 0.1 everywhere
Vocabulary
WordPiece, 30,000 units (30,522 in the uncased release); max length 512
Data
BooksCorpus (800M words) + English Wikipedia text passages (2,500M words); document-level, so long contiguous spans can be sampled
Pre-training
1,000,000 steps of 256 sequences × 512 tokens (128,000 tokens per batch), about 40 epochs over 3.3B words; 90% of steps at length 128, the last 10% at 512 to learn the positional embeddings
Optimiser
Adam, lr 10−410^{-4}, β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, L2 weight decay 0.01, 10,000 warm-up steps then linear decay
Hardware
base: 4 Cloud TPUs (16 chips); large: 16 Cloud TPUs (64 chips); 4 days each
Fine-tuning
batch 16 or 32; lr 5⋅10−55\cdot10^{-5}, 3⋅10−53\cdot10^{-5} or 2⋅10−52\cdot10^{-5}; 2–4 epochs; at most an hour on one Cloud TPU for every task in the paper
IsNext pair · 15 tokens · 2 MLM targets (round(15·0.15))segment A (E_A)[CLS]themanwent[MASK]maskedtothestore[SEP]segment B (E_B) · the real next sentenceheboughtagallonofmilk[MASK]masked[SEP]expected per 512 tokens: 76.8 targets = 61.4 [MASK] + 7.7 random + 7.7 keptuntouched 85.0% · [MASK] 12.0% · random 1.50% · kept 1.50%a visible (non-[MASK]) token is a hidden target with p = 3.41% and is wrong with p = 1.70%[MASK] never appears at fine-tuning; 12.0% of pre-training inputs are [MASK]BERT-base: L = 12, H = 768, A = 12, F = 3072, V = 30,522 → 109,482,240 parameterstoken 23.44M (21.4%) · pos+seg+LN 0.40M · encoder 85.05M (77.7%) · pooler 0.59Mone layer 7,087,872 = attention 2,362,368 + FFN 4,722,432 + 2 LayerNorm 3,072pre-training heads (not in total): MLM 622,650 (decoder tied to the token table), NSP 1,538rule of thumb 12·L·H² + V·H = 108.38M (exact 109.48M)
NSP pairmodel
fig.Top: the MLM corruption of the reference data generator applied to a 15-token pair. At 15% it picks round(15 · 0.15) = 2 non-special positions. Draw 0 masks "went" and "milk". Draw 2 keeps "he" unchanged as a target, and draw 3 turns "a" into the random piece "##ly" while keeping "gallon". The middle band is the exact expectation per 512 tokens: 76.8 targets = 61.4 [MASK] + 7.7 random + 7.7 kept, so 12% of inputs are [MASK] and a visible token is still a hidden target with p = 3.41% and wrong with p = 1.70%. At 100% [MASK] both probabilities fall to 0, so nothing forces the model to model visible tokens. At 0% they become 15% and 7.5%. Bottom: an exact parameter count. BERT-base has 109,482,240 parameters, with the token table at 23.44M (21.4%) and the encoder at 85.05M (77.7%; 7,087,872 per layer). BERT-large has 335,141,888, and its table falls to 9.3%. The rule of thumb 12·L·H² + V·H gives 108.38M for base.

Fine-tuning: one extra layer per task

Fine-tuning changes the inputs and adds one output layer. Single sentences use segment A only. Sentence pairs (entailment, paraphrase) use A and B. Question answering puts the question in A and the passage in B. For classification, a weight matrix W∈RK×HW\in\mathbb R^{K\times H} is added and the loss is log⁡softmax(CW⊤)\log\mathrm{softmax}(CW^{\top}). Token-level tasks such as NER put a classifier on each TiT_i. For SQuAD the only new parameters are a start vector SS and an end vector EE in RH\mathbb R^H.

Pistart=eS⋅Ti∑jeS⋅Tj,score(i,j)=S⋅Ti+E⋅Tj  (j≥i),s^null=S⋅C+E⋅CP^{\text{start}}_i=\frac{e^{S\cdot T_i}}{\sum_j e^{S\cdot T_j}},\qquad \text{score}(i,j)=S\cdot T_i+E\cdot T_j\ \ (j\ge i),\qquad \hat s_{\text{null}}=S\cdot C+E\cdot C
TiT_i
final hidden state of passage token ii
score(i,j)\text{score}(i,j)
the predicted answer is the highest-scoring span with j ≥ i
s^null\hat s_{\text{null}}
SQuAD 2.0: "no answer" is the span at [CLS]; predict an answer only if the best span beats s^null\hat s_{\text{null}} by a threshold τ\tau tuned on dev

Worked example: masking budget and span extraction

Masking one 128-token pre-training sequence

  1. 1The reference generator predicts round(128⋅0.15)=19\mathrm{round}(128\cdot0.15)=19 positions (capped at max_predictions_per_seq = 20 for length 128), never [CLS] or [SEP].
  2. 2Expected rewrites: 19⋅0.8=15.219\cdot0.8=15.2 become [MASK], 1.91.9 become random tokens and 1.91.9 stay unchanged. All 19 are in the loss.
  3. 3The other 109 positions contribute nothing to LMLM\mathcal L_{\text{MLM}} directly. Their only job is to be context.
  4. 4Under the same 128 tokens, a left-to-right LM gets 127 targets. That is the 6.7× difference in supervision per sequence that MLM trades for bidirectionality.

Now fine-tuning on SQuAD. The question is "Where did the cat sit?" and the passage is the cat sat on the mat today. Suppose the fine-tuned model gives these start logits S⋅TiS\cdot T_i and end logits E⋅TjE\cdot T_j.

itokenS·TᵢP_startE·TᵢP_end
0the0.10.073−0.60.030
1cat0.40.0990.30.075
2sat−0.20.054−0.10.050
3on2.10.542−0.30.041
4the0.90.1630.20.068
5mat−0.40.0442.40.612
6today−1.00.0240.80.124
Start and end scores for the passage tokens, with their softmax probabilities (computed in numpy)

Decoding and the training loss

  1. 1Best span: maximise S⋅Ti+E⋅TjS\cdot T_i+E\cdot T_j over j≥ij\ge i. The winner is i=3i=3, j=5j=5 with 2.1+2.4=4.52.1+2.4=4.5, i.e. on the mat. The runners-up are "the mat" (3.3), "on the mat today" (2.9) and "cat sat on the mat" (2.8).
  2. 2Start softmax: log⁡∑jeS⋅Tj=2.713\log\sum_j e^{S\cdot T_j}=2.713, so P3start=e2.1−2.713=0.542P^{\text{start}}_3=e^{2.1-2.713}=0.542. End softmax: log⁡∑jeE⋅Tj=2.891\log\sum_j e^{E\cdot T_j}=2.891, so P5end=e2.4−2.891=0.612P^{\text{end}}_5=e^{2.4-2.891}=0.612.
  3. 3If the gold span is (3, 5), the training loss is −log⁡0.542−log⁡0.612=0.613+0.491=1.105-\log0.542-\log0.612=0.613+0.491=1.105. The paper's objective is this sum of the correct start and end log-likelihoods.
  4. 4The whole task head is SS and EE: 2H=1,5362H=1{,}536 new parameters on BERT-base, against 109.5M fine-tuned.

Results

BERT-large set new results on eleven tasks. On the official GLUE leaderboard it scored 80.5, 7.7 points above the previous best. In the paper's Table 1 average (which excludes WNLI), BERT-base reached 79.6 and BERT-large 82.1, against 75.1 for OpenAI GPT. The base model has the same size as GPT, so most of that 4.5-point gap is the bidirectional objective. MultiNLI accuracy reached 86.7% (4.6 points absolute). SQuAD v1.1 test F1 reached 93.2 (an ensemble also trained on TriviaQA), and SQuAD v2.0 test F1 reached 83.1, 5.1 points above the previous system.

Scale helped even on small datasets. Going from 3 to 24 layers improved every GLUE task tried, including MRPC with 3,600 training examples. The features also worked without fine-tuning. On CoNLL-2003 NER, concatenating the top four hidden layers of BERT-base as fixed features gave 96.1 dev F1, against 96.4 with full fine-tuning.

python
1import numpy as np
2
3CLS, SEP, MASK, PAD = 101, 102, 103, 0 # ids in bert-base-uncased's 30,522-token vocab
4VOCAB = 30522
5
6def mask_tokens(ids, rng, rate=0.15, p_mask=0.8, p_random=0.1):
7 """BERT MLM corruption. ids: (B, T) int array. Returns inputs and labels (-100 = no loss)."""
8 ids = np.asarray(ids)
9 special = np.isin(ids, [CLS, SEP, PAD])
10 chosen = (rng.random(ids.shape) < rate) & ~special # 15% of real tokens become targets
11 labels = np.where(chosen, ids, -100)
12
13 inputs = ids.copy()
14 u = rng.random(ids.shape)
15 to_mask = chosen & (u < p_mask) # 80% -> [MASK]
16 to_rand = chosen & (u >= p_mask) & (u < p_mask + p_random) # 10% -> random token
17 inputs[to_mask] = MASK # remaining 10% stay unchanged
18 inputs[to_rand] = rng.integers(1000, VOCAB, to_rand.sum()) # skip [unused]/special ids
19 return inputs, labels
20
21rng = np.random.default_rng(0)
22body = rng.integers(1000, VOCAB, size=(4096, 126)) # 4,096 fake sequences of 128
23ids = np.concatenate([np.full((4096, 1), CLS), body, np.full((4096, 1), SEP)], axis=1)
24x, y = mask_tokens(ids, rng)
25
26t = y != -100
27print(f"targets {t.mean():.4f} of positions, {t.sum(1).mean():.1f} per sequence") # 0.1471, 18.8
28print(f"[MASK] {(x[t] == MASK).mean():.3f} of targets") # 0.798
29print(f"random {((x[t] != MASK) & (x[t] != ids[t])).mean():.3f}") # 0.103
30print(f"kept {(x[t] == ids[t]).mean():.3f}") # 0.099
31print(f"P(target | visible token) = {t[x != MASK].mean():.4f}") # 0.0336
32
33# With Hugging Face the same corruption is DataCollatorForLanguageModeling(tok, mlm_probability=0.15),
34# and a fine-tuning head is one line, e.g. BertForQuestionAnswering.from_pretrained("bert-base-uncased").
BERT's 80/10/10 MLM corruption in numpy, checked on 4,096 synthetic 128-token sequences (run; outputs shown)

Use when

  • +Deep bidirectional context at every layer; GLUE 80.5 and SQuAD v1.1 F1 93.2 at release
  • +One pre-trained model fine-tunes to classification, tagging, pairs and span QA with a single extra layer
  • +Fine-tuning is cheap: 2–4 epochs, at most an hour on one Cloud TPU for each task in the paper
  • +Base model (110M) is small enough to serve at scale for classification and embeddings

Avoid when

  • −Only about 15% of positions give a training signal, so pre-training is less sample-efficient than next-token LM
  • −[MASK] appears in pre-training but never at fine-tuning (pre-train/fine-tune mismatch)
  • −Masked tokens are predicted independently, so it cannot generate text or model joint dependencies between targets
  • −512-token limit from learned positional embeddings; NSP turned out to be of little use (RoBERTa, ALBERT)

Where it's used

Sentence and pair classification

Sentiment, NLI, paraphrase and intent: a softmax on CC, fine-tuned end to end.

Token tagging

NER, POS and slot filling with a classifier on every final hidden state.

Extractive QA

SQuAD-style span prediction with start/end vectors SS and EE.

Search and retrieval

Cross-encoder re-ranking of query–passage pairs, and the backbone of many dense-embedding models.

Check yourself

Why does BERT replace only 80% of the chosen tokens with [MASK], keeping 10% unchanged and swapping 10% for random tokens?

Check yourself

In BERT's own ablation on BERT-base, which change hurt SQuAD v1.1 dev F1 most (baseline 88.5)?

Remember

  • ✓BERT = a post-LN Transformer encoder with no causal mask: base 12×76812\times768, 12 heads, 110M; large 24×102424\times1024, 16 heads, 340M; 30K WordPiece, 512 positions.
  • ✓Input =Etok+Eseg+Epos=E_{\text{tok}}+E_{\text{seg}}+E_{\text{pos}} with [CLS] first and [SEP] after each segment; C=h[CLS](L)C=h^{(L)}_{\text{[CLS]}} summarises the sequence.
  • ✓MLM picks 15% of tokens and rewrites them 80% [MASK], 10% random, 10% unchanged. All picked positions are scored with a tied-softmax cross-entropy.
  • ✓NSP (50% real next segment, 50% random) is a binary classifier on C. Later work found it of little use.
  • ✓Fine-tuning adds one layer: softmax(CW⊤)\mathrm{softmax}(CW^\top) for classification, per-token classifiers for tagging, and start/end vectors S,ES,E for span QA.
  • ✓Results: GLUE 80.5 (+7.7), MultiNLI 86.7%, SQuAD v1.1 test F1 93.2, SQuAD v2.0 test F1 83.1.
← back to Deep Learning roadmap