Definition
BERT (Bidirectional Encoder Representations from Transformers) is the pre-trained language representation of Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (NAACL 2019). It is a Transformer encoder stack with no causal mask, pre-trained on unlabeled text with two objectives. The masked language model (MLM) hides 15% of the input tokens and predicts them from both sides. Next-sentence prediction (NSP) classifies whether segment B really followed segment A. For each downstream task, one small output layer is added and every weight is fine-tuned.
Before BERT, transfer learning in NLP came in two forms. Feature-based methods such as ELMo (Peters et al., 2018) trained a left-to-right and a right-to-left LSTM language model separately and fed their concatenated states into a task-specific architecture. Fine-tuning methods such as OpenAI GPT (Radford et al., 2018) pre-trained a left-to-right transformer and fine-tuned all of it. Both were limited by the language-model objective. A left-to-right model predicts token from tokens only, so its representation of a word never conditions on the words after it. ELMo's two directions are only joined shallowly, after each has been trained alone.
You cannot simply train a bidirectional encoder on next-token prediction: in a multi-layer bidirectional stack every token can attend to itself, so the target leaks and the task becomes trivial. BERT's answer is the cloze task (Taylor, 1953). Corrupt some input positions and ask the model to reconstruct them. Every layer can now attend in both directions, because the answers are not in the input. BERT-base was deliberately sized like GPT (12 layers, 768 wide, 12 heads) so that the comparison isolates the objective and the attention direction.
Input representation
Every input is one or two segments of contiguous text packed into at most 512 WordPiece tokens. The first token is always [CLS], and each segment ends with [SEP]. A segment is a span of text, not necessarily a linguistic sentence. The vocabulary has 30,000 WordPiece units (30,522 entries in the released uncased checkpoint); see WordPiece.
- WordPiece embedding, ; reused (tied) as the MLM output matrix
- segment embedding: for
[CLS]through the first[SEP], after it - learned absolute position embedding (not sinusoidal)
- the final hidden state of
[CLS], used as the aggregate sequence representation
Masked language model
The data generator picks 15% of the WordPiece positions in each sequence as prediction targets. Each picked position is then rewritten. With probability 0.8 it becomes [MASK], with 0.1 a random token, and with 0.1 it is left unchanged. The loss is cross-entropy over the vocabulary at the picked positions only. The paper's example is my dog is hairy with hairy picked. It becomes my dog is [MASK] 80% of the time, my dog is apple 10% of the time, and stays my dog is hairy the other 10%.
- the picked positions; is the original token there, whatever the input shows
- the MLM transform: dense , GELU, LayerNorm
- row of (tied output embedding); is a per-token output bias
Why not always use [MASK]? The symbol never appears at fine-tuning time, so a model that only learns to predict at [MASK] positions has no reason to build a good representation of a visible token. With the 80/10/10 rule, any visible token might be a target: with , a non-[MASK] token is a hidden target with probability . The model therefore has to keep a contextual estimate of every token. Random replacement touches only of all tokens, which the authors judged too little to hurt language understanding. Their appendix ablation found the mixed strategy best. Always-[MASK] was noticeably worse for the feature-based NER setting, and always-random was worse than both.
The price is sample efficiency. A left-to-right model gets a training signal from every position, and BERT gets one from about 15%. The paper's Figure 5 shows that MLM does converge more slowly than a left-to-right model in pre-training steps, but it overtakes the left-to-right model in accuracy almost immediately.
Next-sentence prediction
Question answering and natural language inference depend on the relation between two texts, which a language model does not learn directly. For each pre-training example, segment B is the actual next segment 50% of the time (IsNext) and a random segment from the corpus otherwise (NotNext). A classifier on predicts the label. The final model reaches 97–98% NSP accuracy. The total pre-training loss is the sum of the mean MLM likelihood and the mean NSP likelihood.
- the pooler, a dense layer on
[CLS]; it ships with the checkpoint - the binary NSP classifier, discarded after pre-training
NSP was later questioned. RoBERTa found that dropping it and packing full sentences from contiguous text matched or beat BERT, and ALBERT replaced it with sentence-order prediction, arguing that NSP is mostly topic detection. In BERT's own ablation on BERT-base, removing NSP lowered QNLI dev accuracy from 88.4 to 84.9, MNLI-m from 84.4 to 83.9 and SQuAD v1.1 F1 from 88.5 to 87.9. Replacing MLM with a left-to-right objective (also without NSP) cost far more: MRPC 86.7 → 77.5 and SQuAD F1 → 77.8.
Configurations and training (Devlin et al. 2019)
- BERT-base
- layers, , heads, FFN , 110M parameters (109,482,240 counted with the pooler)
- BERT-large
- , , , FFN 4096, 340M parameters (335,141,888 counted)
- Block
- post-LN encoder block (as in Vaswani et al.), GELU activation, dropout 0.1 everywhere
- Vocabulary
- WordPiece, 30,000 units (30,522 in the uncased release); max length 512
- Data
- BooksCorpus (800M words) + English Wikipedia text passages (2,500M words); document-level, so long contiguous spans can be sampled
- Pre-training
- 1,000,000 steps of 256 sequences × 512 tokens (128,000 tokens per batch), about 40 epochs over 3.3B words; 90% of steps at length 128, the last 10% at 512 to learn the positional embeddings
- Optimiser
- Adam, lr , , , L2 weight decay 0.01, 10,000 warm-up steps then linear decay
- Hardware
- base: 4 Cloud TPUs (16 chips); large: 16 Cloud TPUs (64 chips); 4 days each
- Fine-tuning
- batch 16 or 32; lr , or ; 2–4 epochs; at most an hour on one Cloud TPU for every task in the paper
Fine-tuning: one extra layer per task
Fine-tuning changes the inputs and adds one output layer. Single sentences use segment A only. Sentence pairs (entailment, paraphrase) use A and B. Question answering puts the question in A and the passage in B. For classification, a weight matrix is added and the loss is . Token-level tasks such as NER put a classifier on each . For SQuAD the only new parameters are a start vector and an end vector in .
- final hidden state of passage token
- the predicted answer is the highest-scoring span with j ≥ i
- SQuAD 2.0: "no answer" is the span at
[CLS]; predict an answer only if the best span beats by a threshold tuned on dev
Worked example: masking budget and span extraction
Masking one 128-token pre-training sequence
- 1The reference generator predicts positions (capped at
max_predictions_per_seq= 20 for length 128), never[CLS]or[SEP]. - 2Expected rewrites: become
[MASK], become random tokens and stay unchanged. All 19 are in the loss. - 3The other 109 positions contribute nothing to directly. Their only job is to be context.
- 4Under the same 128 tokens, a left-to-right LM gets 127 targets. That is the 6.7× difference in supervision per sequence that MLM trades for bidirectionality.
Now fine-tuning on SQuAD. The question is "Where did the cat sit?" and the passage is the cat sat on the mat today. Suppose the fine-tuned model gives these start logits and end logits .
| i | token | S·Tᵢ | P_start | E·Tᵢ | P_end |
|---|---|---|---|---|---|
| 0 | the | 0.1 | 0.073 | −0.6 | 0.030 |
| 1 | cat | 0.4 | 0.099 | 0.3 | 0.075 |
| 2 | sat | −0.2 | 0.054 | −0.1 | 0.050 |
| 3 | on | 2.1 | 0.542 | −0.3 | 0.041 |
| 4 | the | 0.9 | 0.163 | 0.2 | 0.068 |
| 5 | mat | −0.4 | 0.044 | 2.4 | 0.612 |
| 6 | today | −1.0 | 0.024 | 0.8 | 0.124 |
Decoding and the training loss
- 1Best span: maximise over . The winner is , with , i.e. on the mat. The runners-up are "the mat" (3.3), "on the mat today" (2.9) and "cat sat on the mat" (2.8).
- 2Start softmax: , so . End softmax: , so .
- 3If the gold span is (3, 5), the training loss is . The paper's objective is this sum of the correct start and end log-likelihoods.
- 4The whole task head is and : new parameters on BERT-base, against 109.5M fine-tuned.
Results
BERT-large set new results on eleven tasks. On the official GLUE leaderboard it scored 80.5, 7.7 points above the previous best. In the paper's Table 1 average (which excludes WNLI), BERT-base reached 79.6 and BERT-large 82.1, against 75.1 for OpenAI GPT. The base model has the same size as GPT, so most of that 4.5-point gap is the bidirectional objective. MultiNLI accuracy reached 86.7% (4.6 points absolute). SQuAD v1.1 test F1 reached 93.2 (an ensemble also trained on TriviaQA), and SQuAD v2.0 test F1 reached 83.1, 5.1 points above the previous system.
Scale helped even on small datasets. Going from 3 to 24 layers improved every GLUE task tried, including MRPC with 3,600 training examples. The features also worked without fine-tuning. On CoNLL-2003 NER, concatenating the top four hidden layers of BERT-base as fixed features gave 96.1 dev F1, against 96.4 with full fine-tuning.
1import numpy as np23CLS, SEP, MASK, PAD = 101, 102, 103, 0 # ids in bert-base-uncased's 30,522-token vocab4VOCAB = 3052256def mask_tokens(ids, rng, rate=0.15, p_mask=0.8, p_random=0.1):7 """BERT MLM corruption. ids: (B, T) int array. Returns inputs and labels (-100 = no loss)."""8 ids = np.asarray(ids)9 special = np.isin(ids, [CLS, SEP, PAD])10 chosen = (rng.random(ids.shape) < rate) & ~special # 15% of real tokens become targets11 labels = np.where(chosen, ids, -100)1213 inputs = ids.copy()14 u = rng.random(ids.shape)15 to_mask = chosen & (u < p_mask) # 80% -> [MASK]16 to_rand = chosen & (u >= p_mask) & (u < p_mask + p_random) # 10% -> random token17 inputs[to_mask] = MASK # remaining 10% stay unchanged18 inputs[to_rand] = rng.integers(1000, VOCAB, to_rand.sum()) # skip [unused]/special ids19 return inputs, labels2021rng = np.random.default_rng(0)22body = rng.integers(1000, VOCAB, size=(4096, 126)) # 4,096 fake sequences of 12823ids = np.concatenate([np.full((4096, 1), CLS), body, np.full((4096, 1), SEP)], axis=1)24x, y = mask_tokens(ids, rng)2526t = y != -10027print(f"targets {t.mean():.4f} of positions, {t.sum(1).mean():.1f} per sequence") # 0.1471, 18.828print(f"[MASK] {(x[t] == MASK).mean():.3f} of targets") # 0.79829print(f"random {((x[t] != MASK) & (x[t] != ids[t])).mean():.3f}") # 0.10330print(f"kept {(x[t] == ids[t]).mean():.3f}") # 0.09931print(f"P(target | visible token) = {t[x != MASK].mean():.4f}") # 0.03363233# With Hugging Face the same corruption is DataCollatorForLanguageModeling(tok, mlm_probability=0.15),34# and a fine-tuning head is one line, e.g. BertForQuestionAnswering.from_pretrained("bert-base-uncased").Use when
- +Deep bidirectional context at every layer; GLUE 80.5 and SQuAD v1.1 F1 93.2 at release
- +One pre-trained model fine-tunes to classification, tagging, pairs and span QA with a single extra layer
- +Fine-tuning is cheap: 2–4 epochs, at most an hour on one Cloud TPU for each task in the paper
- +Base model (110M) is small enough to serve at scale for classification and embeddings
Avoid when
- −Only about 15% of positions give a training signal, so pre-training is less sample-efficient than next-token LM
- −[MASK] appears in pre-training but never at fine-tuning (pre-train/fine-tune mismatch)
- −Masked tokens are predicted independently, so it cannot generate text or model joint dependencies between targets
- −512-token limit from learned positional embeddings; NSP turned out to be of little use (RoBERTa, ALBERT)
Where it's used
Sentence and pair classification
Sentiment, NLI, paraphrase and intent: a softmax on , fine-tuned end to end.
Token tagging
NER, POS and slot filling with a classifier on every final hidden state.
Extractive QA
SQuAD-style span prediction with start/end vectors and .
Search and retrieval
Cross-encoder re-ranking of query–passage pairs, and the backbone of many dense-embedding models.
Check yourself
Why does BERT replace only 80% of the chosen tokens with [MASK], keeping 10% unchanged and swapping 10% for random tokens?
Check yourself
In BERT's own ablation on BERT-base, which change hurt SQuAD v1.1 dev F1 most (baseline 88.5)?
Remember
- ✓BERT = a post-LN Transformer encoder with no causal mask: base , 12 heads, 110M; large , 16 heads, 340M; 30K WordPiece, 512 positions.
- ✓Input with
[CLS]first and[SEP]after each segment; summarises the sequence. - ✓MLM picks 15% of tokens and rewrites them 80%
[MASK], 10% random, 10% unchanged. All picked positions are scored with a tied-softmax cross-entropy. - ✓NSP (50% real next segment, 50% random) is a binary classifier on C. Later work found it of little use.
- ✓Fine-tuning adds one layer: for classification, per-token classifiers for tagging, and start/end vectors for span QA.
- ✓Results: GLUE 80.5 (+7.7), MultiNLI 86.7%, SQuAD v1.1 test F1 93.2, SQuAD v2.0 test F1 83.1.