ALBERT

ALBERT (Lan et al., ICLR 2020) shrinks BERT by factorizing the embedding (V×E + E×H with E = 128) and sharing one layer's weights across all layers, and swaps NSP for sentence-order prediction. ALBERT-large has 18M parameters vs BERT-large's 334M, at the same compute per token.

14 min read1009 wordsadvanced0 viewsupdated 2026-10-13

Definition

ALBERT (A Lite BERT) is the parameter-reduced BERT of Lan, Chen, Goodman, Gimpel, Sharma and Soricut, "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations" (ICLR 2020, Google Research and TTI-Chicago). It makes three changes. A factorized embedding parameterization maps the vocabulary to a small size EE and then projects to the hidden size HH. Cross-layer parameter sharing reuses one transformer layer's weights at every depth. Sentence-order prediction (SOP) replaces next-sentence prediction. ALBERT-base has 12M parameters against BERT-base's 108M.

By 2019, scaling BERT up was the reliable way to improve it, and it was running into memory limits. The authors' motivating example was a 1.27B-parameter "BERT-xlarge" (H=2048H=2048). It was harder to train and did worse than BERT-large. They asked where BERT's parameters actually go and whether they all need to be separate. Two observations followed. First, the embedding table is V×HV\times H, so it grows with the hidden size even though a context-independent word vector should need far fewer dimensions than a contextual one. Second, the LL layers have identical shapes, so one set of weights can serve as all of them.

ALBERT is not a cheaper model at inference. Sharing removes stored copies, not computation. Every token still passes through all LL layers. The point is that, at a fixed memory budget, you can afford a much wider network. ALBERT-xxlarge (H=4096H=4096) has fewer parameters than BERT-large and outperforms it in all of the paper's comparisons.

Factorized embedding parameterization

V×H⏟BERT  ⟶  V×E⏟lookup+E×H⏟projection,xi↦Wproj Etok[xi]∈RH,E≪H\underbrace{V\times H}_{\text{BERT}}\;\longrightarrow\;\underbrace{V\times E}_{\text{lookup}}+\underbrace{E\times H}_{\text{projection}},\qquad x_i\mapsto W_{\text{proj}}\,E_{\text{tok}}[x_i]\in\mathbb R^{H},\qquad E\ll H
VV
vocabulary size, 30,000 SentencePiece units in ALBERT
EE
embedding size, 128 in all released ALBERT models
HH
hidden size of the transformer layers: 768, 1024, 2048 or 4096
Wproj∈RH×EW_{\text{proj}}\in\mathbb R^{H\times E}
a learned linear map; position and segment embeddings are also E-dimensional and added before it

The argument is about what each matrix represents. Word-piece embeddings are context-independent, and hidden states are context-dependent. The model's capacity lives in the hidden states, so tying the two sizes (E≡HE\equiv H) wastes parameters. The table is also updated sparsely, because each row receives gradient only when its token appears in the batch. For ALBERT-xxlarge, V⋅H=30,000⋅4096=122.88V\cdot H=30{,}000\cdot4096=122.88M becomes V⋅E+E⋅H=3.84M+0.52M=4.36V\cdot E+E\cdot H=3.84\text{M}+0.52\text{M}=4.36M, which is 28× smaller. The low-rank factorization is the same idea as LoRA, applied to an embedding table instead of a weight update.

1 · factorized embedding (ALBERT-xxlarge: V = 30,000, E = 128, H = 4096)token idone-hot ∈ ℝ^VV×E table30,000×128 = 3.84Me ∈ ℝ^128E×H projection128×4096 = 0.52Mh ∈ ℝ^4096BERT-style V×H table: 30,000×4096 = 122.88M (28× more)2 · cross-layer sharinglayer 1layer 2layer 3⋮layer 12one θ, reusedattn 67.1MFFN 134.2Mstored oncexxlarge: 12 passesthrough the sameweights: params÷12, FLOPs ×13 · sentence-order predictionA, B: consecutive segments of one documentSOP positive[CLS]A[SEP]B[SEP]→ in orderSOP negative: same segments, swapped[CLS]B[SEP]A[SEP]→ swappedBERT's NSP negative: B′ from another doc[CLS]A[SEP]B′[SEP]→ NotNextNSP can be solved by topic alone;SOP keeps the topic fixed, so onlydiscourse coherence tells them apart
fig.ALBERT's three changes to BERT, drawn for ALBERT-xxlarge. (1) The token id looks up a 128-dimensional vector in a 30,000×128 table (3.84M), and a 128×4096 matrix (0.52M) projects it to the hidden size. A BERT-style 30,000×4096 table would be 122.88M. (2) One layer's attention and FFN weights (67.1M + 134.2M at H = 4096) are applied 12 times, so the parameters are divided by 12 and the FLOPs are unchanged. (3) SOP positives are two consecutive segments in order, and negatives are the same two swapped. NSP negatives take B′ from another document, which is often detectable from topic alone.

Cross-layer parameter sharing

h(ℓ)=Layerθ(h(ℓ−1)),ℓ=1,…,L,#params=∣θ∣⏟once instead of L ∣θ∣,∣θ∣≈12H2h^{(\ell)}=\mathrm{Layer}_{\theta}\big(h^{(\ell-1)}\big),\quad \ell=1,\dots,L,\qquad \#\text{params}=\underbrace{|\theta|}_{\text{once}}\ \text{instead of}\ L\,|\theta|,\qquad |\theta|\approx12H^2
θ\theta
one layer's attention (4H24H^2) and FFN (8H28H^2 with F=4HF=4H) weights, plus biases and LayerNorms
all−sharedall-shared
ALBERT's default: attention and FFN shared; the paper also tests sharing only one of them
LL
12 or 24; the forward pass still applies θ\theta LL times

Weight sharing across depth had been tried before. The Universal Transformer (Dehghani et al., 2019) shares weights across depth for sequence-to-sequence tasks. ALBERT applies it to pre-training. In the ablation at ALBERT-base size, most of the accuracy lost to sharing came from the FFN parameters. Sharing only attention cost little or nothing. The authors still share everything, because the parameter savings are much larger. The paper also observed a side effect: with sharing, the L2 distance and cosine similarity between each layer's input and output embeddings change smoothly from layer to layer, instead of oscillating as in BERT. Sharing acts as a stabiliser on the network.

Sentence-order prediction

RoBERTa and others had found NSP unhelpful. The ALBERT authors' diagnosis is that NSP mixes topic prediction with coherence prediction. A NotNext pair takes segment B from a different document, so it usually changes topic, and topic is easy to read off with the MLM's own features. SOP keeps the topic fixed. A positive is two consecutive segments from one document in their original order. A negative is the same two segments swapped. Only discourse-level coherence separates them.

L=LMLM+LSOP,LSOP=−log⁡softmax(W pool(C))y,y={1[CLS] A [SEP] B [SEP]0[CLS] B [SEP] A [SEP]\mathcal L=\mathcal L_{\text{MLM}}+\mathcal L_{\text{SOP}},\qquad \mathcal L_{\text{SOP}}=-\log\mathrm{softmax}\big(W\,\mathrm{pool}(C)\big)_{y},\quad y=\begin{cases}1&[\text{CLS}]\,A\,[\text{SEP}]\,B\,[\text{SEP}]\\0&[\text{CLS}]\,B\,[\text{SEP}]\,A\,[\text{SEP}]\end{cases}
A, BA,\,B
two consecutive segments of one document, 50/50 kept in order or swapped
LMLM\mathcal L_{\text{MLM}}
BERT's masked LM, here with n-gram masking: spans of up to 3 tokens, with length nn chosen with probability ∝1/n\propto1/n

The intrinsic test makes the point (Table 5, ALBERT-base). A model trained with NSP solves NSP but scores only 52.0% on SOP, which is chance. It never learned order. A model trained with SOP scores 86.5% on SOP and still 78.9% on NSP. SOP subsumes most of NSP, and the SOP model also did better on the downstream multi-sentence tasks.

Configurations (Lan et al. 2020, Table 1) and training

ALBERT-base
L=12L=12, H=768H=768, E=128E=128, all-shared: 12M parameters (BERT-base: 108M)
ALBERT-large
L=24L=24, H=1024H=1024, E=128E=128: 18M (BERT-large: 334M)
ALBERT-xlarge
L=24L=24, H=2048H=2048, E=128E=128: 60M
ALBERT-xxlarge
L=12L=12, H=4096H=4096, E=128E=128, 64 heads, FFN 16,384: 235M in the paper's table. A 24-layer version was not better and cost more
Vocabulary
30,000 SentencePiece units (as in XLNet), max length 512, with shorter sequences generated 10% of the time
Pre-training
BooksCorpus + English Wikipedia (about 16GB) for the comparisons; MLM with n-gram masking (n≤3n\le3) + SOP; batch 4096, LAMB optimizer, lr 0.00176, 125,000 steps, on 64 to 512 Cloud TPU v3 chips
Final models
trained longer, on additional data (the XLNet and RoBERTa corpora), and with dropout removed, which the authors found improved MLM accuracy
Results
ALBERT-xxlarge ensemble: GLUE 89.4, SQuAD 2.0 test F1 92.2, RACE test accuracy 89.4, all new state of the art at publication
ALBERT-base: L=12, H=768, E=128, share all → 11.68M (paper 12M)stored parameters, this configurationsame L, H with E = H and no sharing: 109.08M (9.3× more)V×E table · positions, segments, E×H projection · attention · FFN · LayerNorm + poolerembedding: V·E + E·H = 3,840,000 + 98,304 = 3.94M vs V·H = 23.04M (5.9× smaller)layers: attention 2.36M + FFN 4.72M per layer, stored once → 7.09Membeddings are 34.3% of the model (unfactorized and unshared: 21.5%)forward compute ≈ 0.170 GFLOP/token = 0.28× BERT-large; sharing does not change it(2 FLOPs per weight in L·(4H² + 2HF) + E·H; excludes attention scores and the MLM head)fp32 weights: 45 MiB vs 416 MiB unfactorized and unsharedtotal parameters vs E (log–log), same L, H, sharing641282565127689.68M31.11MEpaper (Table 1, V = 30K):A-base 12M · ours 11.68MA-large 18M · ours 17.68MA-xlarge 60M · ours 58.72MA-xxlarge 235M · ours 222.60MB-base 108M · ours 109.08MB-large 334M · ours 334.61M
preset
share
V = 30,000 SentencePiece tokens, 512 positions, F = 4H; E is capped at H.
fig.An exact parameter calculator (V = 30,000, 512 positions, F = 4H, biases and LayerNorms counted). The default is ALBERT-base: 11.68M stored parameters, against 109.08M for the same shape with E = H and no sharing (9.3× more). The embedding is V·E + E·H = 3,840,000 + 98,304 instead of V·H = 23.04M, and one 7.09M layer is stored once instead of 12 times. Compute is 0.170 GFLOP per token either way, so sharing saves memory, not time. Our counts are 17.68M for ALBERT-large, 58.72M for xlarge and 222.60M for xxlarge, against 18M, 60M and 235M in the paper's Table 1. The xxlarge figure matches the released checkpoint's 223M. ALBERT-xxlarge stores 2/3 of BERT-large's 334.61M but runs 8.0× its FLOPs per token (4.833 vs 0.604 GFLOP). The same xxlarge shape without either reduction would be 2.56B. With sharing switched to none, the E sweep at ALBERT-base size reproduces Table 3's not-shared column: 87.65M, 89.65M, 93.65M and 109.08M for E = 64, 128, 256 and 768.

Worked example: counting ALBERT-base

11,683,584 parameters, from the bottom up

  1. 1Token table V⋅E=30,000⋅128=3,840,000V\cdot E=30{,}000\cdot128=3{,}840{,}000. Positions 512⋅128=65,536512\cdot128=65{,}536, segments 2⋅128=2562\cdot128=256, embedding LayerNorm 256256.
  2. 2Projection E→HE\to H: 128⋅768+768=99,072128\cdot768+768=99{,}072. The embedding block totals 4,005,1204{,}005{,}120, against 23,040,00023{,}040{,}000 for BERT's V⋅HV\cdot H table alone.
  3. 3One layer: attention 4(7682+768)=2,362,3684(768^2+768)=2{,}362{,}368, FFN 768⋅3072+3072+3072⋅768+768=4,722,432768\cdot3072+3072+3072\cdot768+768=4{,}722{,}432, two LayerNorms 3,0723{,}072. That is 7,087,8727{,}087{,}872, stored once.
  4. 4Pooler 7682+768=590,592768^2+768=590{,}592. Total 4,005,120+7,087,872+590,592=11,683,584≈124{,}005{,}120+7{,}087{,}872+590{,}592=11{,}683{,}584\approx12M, the paper's figure.
  5. 5Without sharing the 12 layers cost 85,054,46485{,}054{,}464, giving 89.6589.65M (the paper's 89M row). With E=HE=H as well, the total is 109.08109.08M, BERT-base with ALBERT's vocabulary.
Eall-shared (paper)all-shared (ours)not-shared (paper)not-shared (ours)
6410M9.68M87M87.65M
12812M11.68M89M89.65M
25616M15.69M93M93.65M
76831M31.11M108M109.08M
Parameters vs embedding size E at ALBERT-base shape (L = 12, H = 768). Paper values from Table 3; our exact counts from the calculator above

Going from E=768E=768 to E=128E=128 saves 19.4M in the all-shared model, 62% of its size at E=768E=768. In the not-shared model it saves the same 19.4M, but that is under a fifth of the total. Factorization matters most once sharing has removed the layer copies. The paper picked E=128E=128 as the best trade-off under sharing, and accuracy was similar across the EE values tested.

python
1import random, torch, torch.nn as nn
2
3class AlbertEncoder(nn.Module):
4 def __init__(self, V=30000, E=128, H=768, L=12, heads=12, max_len=512):
5 super().__init__()
6 self.tok = nn.Embedding(V, E) # V×E instead of V×H
7 self.pos = nn.Embedding(max_len, E)
8 self.seg = nn.Embedding(2, E)
9 self.norm = nn.LayerNorm(E)
10 self.proj = nn.Linear(E, H) # E×H projection into the hidden size
11 self.layer = nn.TransformerEncoderLayer( # ONE post-LN layer, reused L times
12 H, heads, dim_feedforward=4 * H, activation="gelu", batch_first=True, dropout=0.0)
13 self.L = L
14 self.pooler = nn.Sequential(nn.Linear(H, H), nn.Tanh())
15
16 def forward(self, ids, segs, pad_mask=None):
17 T = ids.shape[1]
18 x = self.tok(ids) + self.pos(torch.arange(T, device=ids.device)) + self.seg(segs)
19 h = self.proj(self.norm(x))
20 for _ in range(self.L): # same weights every pass: params ÷ L, FLOPs × 1
21 h = self.layer(h, src_key_padding_mask=pad_mask)
22 return h, self.pooler(h[:, 0]) # token states, pooled [CLS] for SOP
23
24def make_sop(doc, rng=random):
25 """doc: list of token-id segments. Returns (A, B, label): 1 = in order, 0 = swapped."""
26 i = rng.randrange(len(doc) - 1)
27 a, b = doc[i], doc[i + 1] # two CONSECUTIVE segments, same topic
28 return (a, b, 1) if rng.random() < 0.5 else (b, a, 0)
29
30model = AlbertEncoder()
31n = sum(p.numel() for p in model.parameters())
32print(f"{n:,}") # 11,683,584 = 4,005,120 embedding + 7,087,872 shared layer + 590,592 pooler
33# AlbertEncoder(H=4096, L=12, heads=64): 222,595,584 -- the released albert-xxlarge has about 223M
An ALBERT-style encoder in PyTorch: factorized embedding, one shared layer applied L times, and SOP example construction (written for PyTorch 2; not run here, and the parameter counts in the comments were computed separately)

Use when

  • +9–19× fewer stored parameters than BERT at the same shape (ALBERT-base 12M, ALBERT-large 18M)
  • +Makes very wide models (H = 4096) fit in memory: ALBERT-xxlarge beats BERT-large with fewer parameters
  • +SOP is a harder, more useful auxiliary task than NSP
  • +Smaller checkpoints and less optimizer state, so less to communicate in data-parallel training

Avoid when

  • −No inference speedup: every layer still runs, and xxlarge costs about 8× BERT-large's FLOPs per token
  • −All-shared models lose some accuracy at equal shape (ALBERT-base is below BERT-base)
  • −Best results need the very wide xxlarge, which is slow to train and serve
  • −Same 512-token limit and encoder-only design as BERT

Check yourself

ALBERT-large shares all layer weights. Compared with BERT-large (same L = 24, H = 1024), what happens to inference FLOPs per token?

Check yourself

Why does an NSP-trained model score near chance (52.0%) on sentence-order prediction?

Remember

  • ✓Factorized embedding: V×H→V×E+E×HV\times H\to V\times E+E\times H with E=128E=128. At H=4096H=4096 that is 122.88M → 4.36M.
  • ✓Cross-layer sharing: one layer's ≈12H2\approx12H^2 weights are reused at all LL depths. Stored parameters fall about LL-fold, and compute does not change.
  • ✓SOP: positives are consecutive segments in order and negatives the same pair swapped. This removes NSP's topic shortcut (NSP model 52.0% on SOP; SOP model 78.9% on NSP).
  • ✓Configs: base 12M (12×768), large 18M (24×1024), xlarge 60M (24×2048), xxlarge 235M (12×4096, E = 128), vs BERT-base 108M and BERT-large 334M.
  • ✓ALBERT-xxlarge set GLUE 89.4, SQuAD 2.0 F1 92.2 and RACE 89.4, but it is about 3× slower to train than BERT-large: it saves memory, not time.
← back to Deep Learning roadmap