# ALBERT

> ALBERT (Lan et al., ICLR 2020) shrinks BERT by factorizing the embedding (V×E + E×H with E = 128) and sharing one layer's weights across all layers, and swaps NSP for sentence-order prediction. ALBERT-large has 18M parameters vs BERT-large's 334M, at the same compute per token.

Source: https://www.mlfoundry.in/learn/albert  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 14 min read · updated 2026-10-13

**ALBERT:** (A Lite BERT) is the parameter-reduced **BERT** of Lan, Chen, Goodman, Gimpel, Sharma and Soricut, "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations" (ICLR 2020, Google Research and TTI-Chicago). It makes three changes. A **factorized embedding parameterization** maps the vocabulary to a small size $E$ and then projects to the hidden size $H$. **Cross-layer parameter sharing** reuses one transformer layer's weights at every depth. **Sentence-order prediction** (SOP) replaces next-sentence prediction. ALBERT-base has 12M parameters against BERT-base's 108M.

By 2019, scaling BERT up was the reliable way to improve it, and it was running into memory limits. The authors' motivating example was a 1.27B-parameter "BERT-xlarge" ($H=2048$). It was harder to train and did *worse* than BERT-large. They asked where BERT's parameters actually go and whether they all need to be separate. Two observations followed. First, the embedding table is $V\times H$, so it grows with the hidden size even though a context-independent word vector should need far fewer dimensions than a contextual one. Second, the $L$ layers have identical shapes, so one set of weights can serve as all of them.

ALBERT is not a cheaper model at inference. Sharing removes stored copies, not computation. Every token still passes through all $L$ layers. The point is that, at a fixed memory budget, you can afford a much wider network. ALBERT-xxlarge ($H=4096$) has fewer parameters than BERT-large and outperforms it in all of the paper's comparisons.

## Factorized embedding parameterization

$$
\underbrace{V\times H}_{\text{BERT}}\;\longrightarrow\;\underbrace{V\times E}_{\text{lookup}}+\underbrace{E\times H}_{\text{projection}},\qquad x_i\mapsto W_{\text{proj}}\,E_{\text{tok}}[x_i]\in\mathbb R^{H},\qquad E\ll H
$$

- $V$: vocabulary size, 30,000 SentencePiece units in ALBERT
- $E$: embedding size, 128 in all released ALBERT models
- $H$: hidden size of the transformer layers: 768, 1024, 2048 or 4096
- $W_{\text{proj}}\in\mathbb R^{H\times E}$: a learned linear map; position and segment embeddings are also E-dimensional and added before it

The argument is about what each matrix represents. Word-piece embeddings are context-independent, and hidden states are context-dependent. The model's capacity lives in the hidden states, so tying the two sizes ($E\equiv H$) wastes parameters. The table is also updated sparsely, because each row receives gradient only when its token appears in the batch. For ALBERT-xxlarge, $V\cdot H=30{,}000\cdot4096=122.88$M becomes $V\cdot E+E\cdot H=3.84\text{M}+0.52\text{M}=4.36$M, which is 28× smaller. The low-rank factorization is the same idea as **LoRA**, applied to an embedding table instead of a weight update.

> Figure: ALBERT's three changes to BERT, drawn for ALBERT-xxlarge. (1) The token id looks up a 128-dimensional vector in a 30,000×128 table (3.84M), and a 128×4096 matrix (0.52M) projects it to the hidden size. A BERT-style 30,000×4096 table would be 122.88M. (2) One layer's attention and FFN weights (67.1M + 134.2M at H = 4096) are applied 12 times, so the parameters are divided by 12 and the FLOPs are unchanged. (3) SOP positives are two consecutive segments in order, and negatives are the same two swapped. NSP negatives take B′ from another document, which is often detectable from topic alone.

## Cross-layer parameter sharing

$$
h^{(\ell)}=\mathrm{Layer}_{\theta}\big(h^{(\ell-1)}\big),\quad \ell=1,\dots,L,\qquad \#\text{params}=\underbrace{|\theta|}_{\text{once}}\ \text{instead of}\ L\,|\theta|,\qquad |\theta|\approx12H^2
$$

- $\theta$: one layer's attention ($4H^2$) and FFN ($8H^2$ with $F=4H$) weights, plus biases and LayerNorms
- $all-shared$: ALBERT's default: attention **and** FFN shared; the paper also tests sharing only one of them
- $L$: 12 or 24; the forward pass still applies $\theta$ $L$ times

Weight sharing across depth had been tried before. The Universal Transformer (Dehghani et al., 2019) shares weights across depth for sequence-to-sequence tasks. ALBERT applies it to pre-training. In the ablation at ALBERT-base size, most of the accuracy lost to sharing came from the **FFN** parameters. Sharing only attention cost little or nothing. The authors still share everything, because the parameter savings are much larger. The paper also observed a side effect: with sharing, the L2 distance and cosine similarity between each layer's input and output embeddings change smoothly from layer to layer, instead of oscillating as in BERT. Sharing acts as a stabiliser on the network.

## Sentence-order prediction

**RoBERTa** and others had found NSP unhelpful. The ALBERT authors' diagnosis is that NSP mixes **topic prediction** with **coherence prediction**. A NotNext pair takes segment B from a different document, so it usually changes topic, and topic is easy to read off with the MLM's own features. SOP keeps the topic fixed. A positive is two consecutive segments from one document in their original order. A negative is the same two segments swapped. Only discourse-level coherence separates them.

$$
\mathcal L=\mathcal L_{\text{MLM}}+\mathcal L_{\text{SOP}},\qquad \mathcal L_{\text{SOP}}=-\log\mathrm{softmax}\big(W\,\mathrm{pool}(C)\big)_{y},\quad y=\begin{cases}1&[\text{CLS}]\,A\,[\text{SEP}]\,B\,[\text{SEP}]\\0&[\text{CLS}]\,B\,[\text{SEP}]\,A\,[\text{SEP}]\end{cases}
$$

- $A,\,B$: two consecutive segments of one document, 50/50 kept in order or swapped
- $\mathcal L_{\text{MLM}}$: BERT's masked LM, here with **n-gram masking**: spans of up to 3 tokens, with length $n$ chosen with probability $\propto1/n$

The intrinsic test makes the point (Table 5, ALBERT-base). A model trained with NSP solves NSP but scores only **52.0%** on SOP, which is chance. It never learned order. A model trained with SOP scores **86.5%** on SOP and still **78.9%** on NSP. SOP subsumes most of NSP, and the SOP model also did better on the downstream multi-sentence tasks.

**Configurations (Lan et al. 2020, Table 1) and training**

- **ALBERT-base:** $L=12$, $H=768$, $E=128$, all-shared: 12M parameters (BERT-base: 108M)
- **ALBERT-large:** $L=24$, $H=1024$, $E=128$: 18M (BERT-large: 334M)
- **ALBERT-xlarge:** $L=24$, $H=2048$, $E=128$: 60M
- **ALBERT-xxlarge:** $L=12$, $H=4096$, $E=128$, 64 heads, FFN 16,384: 235M in the paper's table. A 24-layer version was not better and cost more
- **Vocabulary:** 30,000 SentencePiece units (as in XLNet), max length 512, with shorter sequences generated 10% of the time
- **Pre-training:** BooksCorpus + English Wikipedia (about 16GB) for the comparisons; MLM with n-gram masking ($n\le3$) + SOP; batch 4096, LAMB optimizer, lr 0.00176, 125,000 steps, on 64 to 512 Cloud TPU v3 chips
- **Final models:** trained longer, on additional data (the XLNet and RoBERTa corpora), and with dropout removed, which the authors found improved MLM accuracy
- **Results:** ALBERT-xxlarge ensemble: GLUE 89.4, SQuAD 2.0 test F1 92.2, RACE test accuracy 89.4, all new state of the art at publication

> Figure: An exact parameter calculator (V = 30,000, 512 positions, F = 4H, biases and LayerNorms counted). The default is ALBERT-base: 11.68M stored parameters, against 109.08M for the same shape with E = H and no sharing (9.3× more). The embedding is V·E + E·H = 3,840,000 + 98,304 instead of V·H = 23.04M, and one 7.09M layer is stored once instead of 12 times. Compute is 0.170 GFLOP per token either way, so sharing saves memory, not time. Our counts are 17.68M for ALBERT-large, 58.72M for xlarge and 222.60M for xxlarge, against 18M, 60M and 235M in the paper's Table 1. The xxlarge figure matches the released checkpoint's 223M. ALBERT-xxlarge stores 2/3 of BERT-large's 334.61M but runs 8.0× its FLOPs per token (4.833 vs 0.604 GFLOP). The same xxlarge shape without either reduction would be 2.56B. With sharing switched to none, the E sweep at ALBERT-base size reproduces Table 3's not-shared column: 87.65M, 89.65M, 93.65M and 109.08M for E = 64, 128, 256 and 768.

## Worked example: counting ALBERT-base

**11,683,584 parameters, from the bottom up**

1. Token table $V\cdot E=30{,}000\cdot128=3{,}840{,}000$. Positions $512\cdot128=65{,}536$, segments $2\cdot128=256$, embedding LayerNorm $256$.
2. Projection $E\to H$: $128\cdot768+768=99{,}072$. The embedding block totals $4{,}005{,}120$, against $23{,}040{,}000$ for BERT's $V\cdot H$ table alone.
3. One layer: attention $4(768^2+768)=2{,}362{,}368$, FFN $768\cdot3072+3072+3072\cdot768+768=4{,}722{,}432$, two LayerNorms $3{,}072$. That is $7{,}087{,}872$, **stored once**.
4. Pooler $768^2+768=590{,}592$. Total $4{,}005{,}120+7{,}087{,}872+590{,}592=11{,}683{,}584\approx12$M, the paper's figure.
5. Without sharing the 12 layers cost $85{,}054{,}464$, giving $89.65$M (the paper's 89M row). With $E=H$ as well, the total is $109.08$M, BERT-base with ALBERT's vocabulary.

Parameters vs embedding size E at ALBERT-base shape (L = 12, H = 768). Paper values from Table 3; our exact counts from the calculator above

| E | all-shared (paper) | all-shared (ours) | not-shared (paper) | not-shared (ours) |
| --- | --- | --- | --- | --- |
| 64 | 10M | 9.68M | 87M | 87.65M |
| 128 | 12M | 11.68M | 89M | 89.65M |
| 256 | 16M | 15.69M | 93M | 93.65M |
| 768 | 31M | 31.11M | 108M | 109.08M |

Going from $E=768$ to $E=128$ saves 19.4M in the all-shared model, 62% of its size at $E=768$. In the not-shared model it saves the same 19.4M, but that is under a fifth of the total. Factorization matters most once sharing has removed the layer copies. The paper picked $E=128$ as the best trade-off under sharing, and accuracy was similar across the $E$ values tested.

```python
import random, torch, torch.nn as nn

class AlbertEncoder(nn.Module):
    def __init__(self, V=30000, E=128, H=768, L=12, heads=12, max_len=512):
        super().__init__()
        self.tok = nn.Embedding(V, E)                  # V×E instead of V×H
        self.pos = nn.Embedding(max_len, E)
        self.seg = nn.Embedding(2, E)
        self.norm = nn.LayerNorm(E)
        self.proj = nn.Linear(E, H)                    # E×H projection into the hidden size
        self.layer = nn.TransformerEncoderLayer(       # ONE post-LN layer, reused L times
            H, heads, dim_feedforward=4 * H, activation="gelu", batch_first=True, dropout=0.0)
        self.L = L
        self.pooler = nn.Sequential(nn.Linear(H, H), nn.Tanh())

    def forward(self, ids, segs, pad_mask=None):
        T = ids.shape[1]
        x = self.tok(ids) + self.pos(torch.arange(T, device=ids.device)) + self.seg(segs)
        h = self.proj(self.norm(x))
        for _ in range(self.L):                        # same weights every pass: params ÷ L, FLOPs × 1
            h = self.layer(h, src_key_padding_mask=pad_mask)
        return h, self.pooler(h[:, 0])                 # token states, pooled [CLS] for SOP

def make_sop(doc, rng=random):
    """doc: list of token-id segments. Returns (A, B, label): 1 = in order, 0 = swapped."""
    i = rng.randrange(len(doc) - 1)
    a, b = doc[i], doc[i + 1]                          # two CONSECUTIVE segments, same topic
    return (a, b, 1) if rng.random() < 0.5 else (b, a, 0)

model = AlbertEncoder()
n = sum(p.numel() for p in model.parameters())
print(f"{n:,}")   # 11,683,584 = 4,005,120 embedding + 7,087,872 shared layer + 590,592 pooler
# AlbertEncoder(H=4096, L=12, heads=64): 222,595,584 -- the released albert-xxlarge has about 223M
```

An ALBERT-style encoder in PyTorch: factorized embedding, one shared layer applied L times, and SOP example construction (written for PyTorch 2; not run here, and the parameter counts in the comments were computed separately)

> **Fewer parameters is not faster:** Cross-layer sharing divides stored weights by $L$ and leaves FLOPs unchanged. ALBERT-xxlarge stores fewer weights than BERT-large (235M vs 334M) but does about $8\times$ the matrix multiplies per token ($4.83$ vs $0.60$ GFLOP), and the paper reports it about 3× slower to train. ALBERT-large is about 1.7× faster to train than BERT-large, because there is less to communicate between workers and less optimizer state, but it is also less accurate. For latency-bound inference, reach for **Knowledge distillation** (e.g. **DistilBERT**), **Quantization** or **Pruning** instead.

> **SOP is NSP without the shortcut:** NSP negatives differ from positives in two ways, topic and order, and the easy one (topic) is enough to solve the task. SOP holds topic fixed by construction, so the only signal left is the one the auxiliary loss was meant to teach. An NSP-trained model is at chance on SOP (52.0%). An SOP-trained model still gets 78.9% on NSP.

> **Where ALBERT fits now:** ALBERT's ideas outlived the model. Factorized and tied embeddings, and depth-wise weight sharing, show up in small and on-device models where memory, not compute, is the limit. For most fine-tuning work, RoBERTa-class encoders or distilled models gave a better accuracy-per-FLOP trade, since ALBERT's savings are in memory. See **Encoder-only models**.

**Strengths**

- 9–19× fewer stored parameters than BERT at the same shape (ALBERT-base 12M, ALBERT-large 18M)
- Makes very wide models (H = 4096) fit in memory: ALBERT-xxlarge beats BERT-large with fewer parameters
- SOP is a harder, more useful auxiliary task than NSP
- Smaller checkpoints and less optimizer state, so less to communicate in data-parallel training

**Limitations**

- No inference speedup: every layer still runs, and xxlarge costs about 8× BERT-large's FLOPs per token
- All-shared models lose some accuracy at equal shape (ALBERT-base is below BERT-base)
- Best results need the very wide xxlarge, which is slow to train and serve
- Same 512-token limit and encoder-only design as BERT

**Question:** ALBERT-large shares all layer weights. Compared with BERT-large (same L = 24, H = 1024), what happens to inference FLOPs per token?

- A. They fall by about 24×, like the layer parameters
- B. They fall by about 19×, like the total parameters
- C. They stay essentially the same, because all 24 layers still run
- D. They double, because of the extra E×H projection

**Answer:** C. They stay essentially the same, because all 24 layers still run — Sharing reuses one $\theta$ at all 24 depths, so stored weights shrink (334.6M → 17.7M) but every token still passes through 24 layers. The extra $E\times H$ projection adds $128\cdot1024$ multiply-adds per token, under 0.05% of the 24-layer forward pass.

**Question:** Why does an NSP-trained model score near chance (52.0%) on sentence-order prediction?

- A. Its pooler has the wrong output size
- B. NSP negatives come from another document, so NSP can be solved by topic, and the model never learns to use order
- C. SOP uses a different vocabulary
- D. SOP requires n-gram masking

**Answer:** B. NSP negatives come from another document, so NSP can be solved by topic, and the model never learns to use order — In SOP the two segments are the same, so topic is identical for positives and negatives. A model that learned NSP's topic shortcut has no feature that separates A-then-B from B-then-A. The reverse transfer is much better: an SOP-trained model gets 78.9% on NSP.

**Key points**

- Factorized embedding: $V\times H\to V\times E+E\times H$ with $E=128$. At $H=4096$ that is 122.88M → 4.36M.
- Cross-layer sharing: one layer's $\approx12H^2$ weights are reused at all $L$ depths. Stored parameters fall about $L$-fold, and compute does not change.
- SOP: positives are consecutive segments in order and negatives the same pair swapped. This removes NSP's topic shortcut (NSP model 52.0% on SOP; SOP model 78.9% on NSP).
- Configs: base 12M (12×768), large 18M (24×1024), xlarge 60M (24×2048), xxlarge 235M (12×4096, E = 128), vs BERT-base 108M and BERT-large 334M.
- ALBERT-xxlarge set GLUE 89.4, SQuAD 2.0 F1 92.2 and RACE 89.4, but it is about 3× slower to train than BERT-large: it saves memory, not time.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
