Definition
ALBERT (A Lite BERT) is the parameter-reduced BERT of Lan, Chen, Goodman, Gimpel, Sharma and Soricut, "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations" (ICLR 2020, Google Research and TTI-Chicago). It makes three changes. A factorized embedding parameterization maps the vocabulary to a small size and then projects to the hidden size . Cross-layer parameter sharing reuses one transformer layer's weights at every depth. Sentence-order prediction (SOP) replaces next-sentence prediction. ALBERT-base has 12M parameters against BERT-base's 108M.
By 2019, scaling BERT up was the reliable way to improve it, and it was running into memory limits. The authors' motivating example was a 1.27B-parameter "BERT-xlarge" (). It was harder to train and did worse than BERT-large. They asked where BERT's parameters actually go and whether they all need to be separate. Two observations followed. First, the embedding table is , so it grows with the hidden size even though a context-independent word vector should need far fewer dimensions than a contextual one. Second, the layers have identical shapes, so one set of weights can serve as all of them.
ALBERT is not a cheaper model at inference. Sharing removes stored copies, not computation. Every token still passes through all layers. The point is that, at a fixed memory budget, you can afford a much wider network. ALBERT-xxlarge () has fewer parameters than BERT-large and outperforms it in all of the paper's comparisons.
Factorized embedding parameterization
- vocabulary size, 30,000 SentencePiece units in ALBERT
- embedding size, 128 in all released ALBERT models
- hidden size of the transformer layers: 768, 1024, 2048 or 4096
- a learned linear map; position and segment embeddings are also E-dimensional and added before it
The argument is about what each matrix represents. Word-piece embeddings are context-independent, and hidden states are context-dependent. The model's capacity lives in the hidden states, so tying the two sizes () wastes parameters. The table is also updated sparsely, because each row receives gradient only when its token appears in the batch. For ALBERT-xxlarge, M becomes M, which is 28× smaller. The low-rank factorization is the same idea as LoRA, applied to an embedding table instead of a weight update.
Cross-layer parameter sharing
- one layer's attention () and FFN ( with ) weights, plus biases and LayerNorms
- ALBERT's default: attention and FFN shared; the paper also tests sharing only one of them
- 12 or 24; the forward pass still applies times
Weight sharing across depth had been tried before. The Universal Transformer (Dehghani et al., 2019) shares weights across depth for sequence-to-sequence tasks. ALBERT applies it to pre-training. In the ablation at ALBERT-base size, most of the accuracy lost to sharing came from the FFN parameters. Sharing only attention cost little or nothing. The authors still share everything, because the parameter savings are much larger. The paper also observed a side effect: with sharing, the L2 distance and cosine similarity between each layer's input and output embeddings change smoothly from layer to layer, instead of oscillating as in BERT. Sharing acts as a stabiliser on the network.
Sentence-order prediction
RoBERTa and others had found NSP unhelpful. The ALBERT authors' diagnosis is that NSP mixes topic prediction with coherence prediction. A NotNext pair takes segment B from a different document, so it usually changes topic, and topic is easy to read off with the MLM's own features. SOP keeps the topic fixed. A positive is two consecutive segments from one document in their original order. A negative is the same two segments swapped. Only discourse-level coherence separates them.
- two consecutive segments of one document, 50/50 kept in order or swapped
- BERT's masked LM, here with n-gram masking: spans of up to 3 tokens, with length chosen with probability
The intrinsic test makes the point (Table 5, ALBERT-base). A model trained with NSP solves NSP but scores only 52.0% on SOP, which is chance. It never learned order. A model trained with SOP scores 86.5% on SOP and still 78.9% on NSP. SOP subsumes most of NSP, and the SOP model also did better on the downstream multi-sentence tasks.
Configurations (Lan et al. 2020, Table 1) and training
- ALBERT-base
- , , , all-shared: 12M parameters (BERT-base: 108M)
- ALBERT-large
- , , : 18M (BERT-large: 334M)
- ALBERT-xlarge
- , , : 60M
- ALBERT-xxlarge
- , , , 64 heads, FFN 16,384: 235M in the paper's table. A 24-layer version was not better and cost more
- Vocabulary
- 30,000 SentencePiece units (as in XLNet), max length 512, with shorter sequences generated 10% of the time
- Pre-training
- BooksCorpus + English Wikipedia (about 16GB) for the comparisons; MLM with n-gram masking () + SOP; batch 4096, LAMB optimizer, lr 0.00176, 125,000 steps, on 64 to 512 Cloud TPU v3 chips
- Final models
- trained longer, on additional data (the XLNet and RoBERTa corpora), and with dropout removed, which the authors found improved MLM accuracy
- Results
- ALBERT-xxlarge ensemble: GLUE 89.4, SQuAD 2.0 test F1 92.2, RACE test accuracy 89.4, all new state of the art at publication
Worked example: counting ALBERT-base
11,683,584 parameters, from the bottom up
- 1Token table . Positions , segments , embedding LayerNorm .
- 2Projection : . The embedding block totals , against for BERT's table alone.
- 3One layer: attention , FFN , two LayerNorms . That is , stored once.
- 4Pooler . Total M, the paper's figure.
- 5Without sharing the 12 layers cost , giving M (the paper's 89M row). With as well, the total is M, BERT-base with ALBERT's vocabulary.
| E | all-shared (paper) | all-shared (ours) | not-shared (paper) | not-shared (ours) |
|---|---|---|---|---|
| 64 | 10M | 9.68M | 87M | 87.65M |
| 128 | 12M | 11.68M | 89M | 89.65M |
| 256 | 16M | 15.69M | 93M | 93.65M |
| 768 | 31M | 31.11M | 108M | 109.08M |
Going from to saves 19.4M in the all-shared model, 62% of its size at . In the not-shared model it saves the same 19.4M, but that is under a fifth of the total. Factorization matters most once sharing has removed the layer copies. The paper picked as the best trade-off under sharing, and accuracy was similar across the values tested.
1import random, torch, torch.nn as nn23class AlbertEncoder(nn.Module):4 def __init__(self, V=30000, E=128, H=768, L=12, heads=12, max_len=512):5 super().__init__()6 self.tok = nn.Embedding(V, E) # V×E instead of V×H7 self.pos = nn.Embedding(max_len, E)8 self.seg = nn.Embedding(2, E)9 self.norm = nn.LayerNorm(E)10 self.proj = nn.Linear(E, H) # E×H projection into the hidden size11 self.layer = nn.TransformerEncoderLayer( # ONE post-LN layer, reused L times12 H, heads, dim_feedforward=4 * H, activation="gelu", batch_first=True, dropout=0.0)13 self.L = L14 self.pooler = nn.Sequential(nn.Linear(H, H), nn.Tanh())1516 def forward(self, ids, segs, pad_mask=None):17 T = ids.shape[1]18 x = self.tok(ids) + self.pos(torch.arange(T, device=ids.device)) + self.seg(segs)19 h = self.proj(self.norm(x))20 for _ in range(self.L): # same weights every pass: params ÷ L, FLOPs × 121 h = self.layer(h, src_key_padding_mask=pad_mask)22 return h, self.pooler(h[:, 0]) # token states, pooled [CLS] for SOP2324def make_sop(doc, rng=random):25 """doc: list of token-id segments. Returns (A, B, label): 1 = in order, 0 = swapped."""26 i = rng.randrange(len(doc) - 1)27 a, b = doc[i], doc[i + 1] # two CONSECUTIVE segments, same topic28 return (a, b, 1) if rng.random() < 0.5 else (b, a, 0)2930model = AlbertEncoder()31n = sum(p.numel() for p in model.parameters())32print(f"{n:,}") # 11,683,584 = 4,005,120 embedding + 7,087,872 shared layer + 590,592 pooler33# AlbertEncoder(H=4096, L=12, heads=64): 222,595,584 -- the released albert-xxlarge has about 223MUse when
- +9–19× fewer stored parameters than BERT at the same shape (ALBERT-base 12M, ALBERT-large 18M)
- +Makes very wide models (H = 4096) fit in memory: ALBERT-xxlarge beats BERT-large with fewer parameters
- +SOP is a harder, more useful auxiliary task than NSP
- +Smaller checkpoints and less optimizer state, so less to communicate in data-parallel training
Avoid when
- −No inference speedup: every layer still runs, and xxlarge costs about 8× BERT-large's FLOPs per token
- −All-shared models lose some accuracy at equal shape (ALBERT-base is below BERT-base)
- −Best results need the very wide xxlarge, which is slow to train and serve
- −Same 512-token limit and encoder-only design as BERT
Check yourself
ALBERT-large shares all layer weights. Compared with BERT-large (same L = 24, H = 1024), what happens to inference FLOPs per token?
Check yourself
Why does an NSP-trained model score near chance (52.0%) on sentence-order prediction?
Remember
- ✓Factorized embedding: with . At that is 122.88M → 4.36M.
- ✓Cross-layer sharing: one layer's weights are reused at all depths. Stored parameters fall about -fold, and compute does not change.
- ✓SOP: positives are consecutive segments in order and negatives the same pair swapped. This removes NSP's topic shortcut (NSP model 52.0% on SOP; SOP model 78.9% on NSP).
- ✓Configs: base 12M (12×768), large 18M (24×1024), xlarge 60M (24×2048), xxlarge 235M (12×4096, E = 128), vs BERT-base 108M and BERT-large 334M.
- ✓ALBERT-xxlarge set GLUE 89.4, SQuAD 2.0 F1 92.2 and RACE 89.4, but it is about 3× slower to train than BERT-large: it saves memory, not time.