# HuBERT

> HuBERT clusters frames offline with k-means (MFCCs, k = 100, then its own layer features, k = 500) and trains a Transformer to predict the cluster id of masked frames. Each round of re-clustering gives better targets.

Source: https://www.mlfoundry.in/learn/hubert  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 8 min read · updated 2026-10-15

**HuBERT:** (Hidden-Unit BERT) is a self-supervised speech encoder trained by **masked prediction of offline cluster labels**. k-means assigns every 20 ms frame a discrete "hidden unit". The model sees the waveform with spans masked and must classify the unit of each masked frame. Introduced by Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov and Mohamed, "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units" (Meta AI, IEEE/ACM TASLP 2021).

The input is raw 16 kHz audio and the output is one vector per 20 ms frame. Speech has no given vocabulary of tokens: unlike text, there is no lexicon to mask and predict, and segment boundaries are unknown. **wav2vec 2.0** solved this by learning a quantiser jointly with the encoder and using a contrastive loss. That needs distractor sampling, a diversity loss and a Gumbel temperature schedule, and the codebook can still collapse.

HuBERT splits the problem in two. Targets are produced **offline** by plain **k-Means**, with no gradients. The model is then trained with an ordinary cross-entropy loss, as in **BERT**. The targets need not be good. They only need to be *consistent*, because the model learns from the masked frames' context, and its own features can then be re-clustered into better targets.

## Targets and model

**Iteration 1.** k-means with $k=100$ on 39-dimensional **MFCC** features (13 coefficients plus first and second deltas). **Iteration 2.** k-means with $k=500$ on the output of layer 6 of the iteration-1 model. The model itself uses the CNN feature encoder of wav2vec 2.0 (stride 320, 20 ms) and a Transformer. Spans are masked as in wav2vec 2.0, with $p=8\%$ of frames chosen as starts and span length $M=10$.

> Figure: HuBERT training. Top: offline target generation. Frame features (MFCCs in iteration 1, a middle Transformer layer later) are clustered by k-means into one unit id per frame. Bottom: the model reads the waveform through the CNN encoder, spans are masked (p = 8%, M = 10) and the Transformer predicts each masked frame's unit with a softmax over learned unit embeddings. Unmasked frames carry no loss. After training, a middle layer is re-clustered for the next iteration (dashed arrow).

$$
p(c\mid\tilde X,t)=\frac{\exp\!\big(\cos(A\,o_t,\,e_c)/\tau\big)}{\sum_{c'=1}^{C}\exp\!\big(\cos(A\,o_t,\,e_{c'})/\tau\big)},\qquad \mathcal L_m=-\sum_{t\in\mathcal M}\log p(z_t\mid\tilde X,t)
$$

- $\tilde X$: the input with frames in the masked set $\mathcal M$ replaced by a learned mask embedding
- $o_t,\;A$: Transformer output at frame $t$ and a learned projection
- $e_c$: learned embedding of unit $c$, one of $C$ clusters; $\tau=0.1$
- $z_t$: k-means label of frame $t$

The full loss is $\alpha\,\mathcal L_m+(1-\alpha)\,\mathcal L_u$, where $\mathcal L_u$ is the same cross-entropy on unmasked frames. The paper found $\alpha=1$ works best: loss on masked frames only. Predicting a visible frame's unit just copies the clustering, errors included. Predicting a hidden frame's unit forces the model to infer it from context, and context averages out the clustering's noise. This is also why noisy MFCC targets work at all.

## Measuring target quality

The paper scores clusterings against forced-aligned phone labels $y$, with three numbers. **Phone purity** $\mathbb E_z[\max_y p(y\mid z)]$: how pure each cluster is. **Cluster purity** $\mathbb E_y[\max_z p(z\mid y)]$: how concentrated each phone is. **PNMI** $=I(y;z)/H(y)$: the share of phone uncertainty that the unit explains. More clusters raise phone purity and lower cluster purity, so PNMI is the fairer single number. Features from the iteration-1 model give much higher PNMI than MFCCs, which is the point of iterating.

> Figure: k-means targets on a toy 600-frame utterance with 6 phones and noisy 2-d features (σ = 0.45). At the default k = 6, iteration 1 (per-frame, MFCC-like features) has phone purity 0.638, cluster purity 0.638 and PNMI 0.442. Iteration 2 averages each frame with its ±2 neighbours, a stand-in for the context a trained model uses, and PNMI rises to 0.553. The unit strip shows over-segmentation: 5 phone segments become 25 units in iteration 1 but 12 in iteration 2. At k = 32, phone purity is 0.622 versus 0.788 and PNMI 0.528 versus 0.714, while cluster purity falls to 0.208 and 0.260. At k = 2, cluster purity is 0.860, yet PNMI is only 0.189.

**Configurations (Hsu et al., 2021)**

- **BASE:** 12 layers, width 768; about 95M parameters; 2 iterations on LibriSpeech 960 h
- **LARGE:** 24 layers, width 1024; about 317M parameters; Libri-Light 60k h
- **X-LARGE:** 48 layers, width 1280; about 964M parameters; Libri-Light 60k h
- **Targets for LARGE / X-LARGE:** one iteration, k = 500 on layer 9 of the second-iteration BASE model
- **Results:** matches or beats wav2vec 2.0 on LibriSpeech and Libri-light fine-tuning splits (10 min to 960 h); X-LARGE gives up to 19% and 13% relative WER reduction on dev-other and test-other

## Compared with wav2vec 2.0

| Aspect | wav2vec 2.0 | HuBERT |
| --- | --- | --- |
| Targets | quantised latents, learned jointly (2 × 320 codebooks) | k-means ids, computed offline (k = 100, then 500) |
| Loss | InfoNCE: target vs 100 distractors | cross-entropy over all C units |
| Extra terms | diversity loss, Gumbel temperature schedule | none |
| Masking | p = 0.065, M = 10 | p = 8%, M = 10 |
| Training | one stage | iterative: train, re-cluster, retrain |

The price of the simpler loss is the extra stages: clustering hundreds of hours of features and retraining from scratch. The gain is stability. The targets are fixed during a training run, so the model cannot make its task easier by changing them, as a jointly learned quantiser can.

## Worked example: purity and PNMI

Ten frames with phones $y$ = a a a a b b b b b b and units $z$ = 1 1 1 2 2 2 3 3 3 3.

Counts and metrics, logs base 2. Computed in Python.

| Quantity | Computation | Value |
| --- | --- | --- |
| joint counts | (a,1) = 3, (a,2) = 1, (b,2) = 2, (b,3) = 4 | 10 frames |
| marginals | p(a) = 0.4, p(b) = 0.6 · p(z) = 0.3, 0.3, 0.4 |  |
| phone purity | (3 + 2 + 4) / 10 (majority phone of each unit) | 0.900 |
| cluster purity | (3 + 4) / 10 (majority unit of each phone) | 0.700 |
| H(y) | −0.4 log 0.4 − 0.6 log 0.6 | 0.971 bits |
| I(y; z) | 0.3 log(0.3/0.12) + 0.1 log(0.1/0.12) + 0.2 log(0.2/0.18) + 0.4 log(0.4/0.24) | 0.695 bits |
| PNMI | 0.695 / 0.971 | 0.716 |

Unit 2 is mixed. It covers the end of a and the start of b, which is typical at phone boundaries. Collapsing repeats gives the unit string 1 2 3, the form that unit language models are trained on. For the prediction itself, suppose $\cos(A o_t,e_c)$ is 0.7, 0.4 and −0.2 for three units and the target is the first. At $\tau=0.1$ the softmax gives it 0.953, a loss of 0.049 nats.

```python
import numpy as np, soundfile as sf, torch
from sklearn.cluster import MiniBatchKMeans
from transformers import AutoFeatureExtractor, HubertModel

fe = AutoFeatureExtractor.from_pretrained("facebook/hubert-base-ls960")
model = HubertModel.from_pretrained("facebook/hubert-base-ls960").eval()

def layer_features(path, layer=6):
    wav, sr = sf.read(path, dtype="float32")                   # mono, 16 kHz
    inputs = fe(wav, sampling_rate=sr, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs, output_hidden_states=True)
    return out.hidden_states[layer][0].numpy()                 # (frames, 768), 50 per second

feats = np.concatenate([layer_features(p) for p in train_files])
km = MiniBatchKMeans(n_clusters=500, batch_size=10_000, n_init=3).fit(feats)

units = km.predict(layer_features("utterance.wav"))            # one unit id per 20 ms frame
dedup = units[np.r_[True, units[1:] != units[:-1]]]            # collapse repeats for unit LMs
```

Extract HuBERT units with Hugging Face transformers and scikit-learn: a middle layer, k-means with 500 clusters, one id per 20 ms. Not run here (no torch/transformers in this environment). train_files is your list of 16 kHz mono WAVs.

- **ASR and SUPERB tasks:** Fine-tuned with **Connectionist Temporal Classification** for recognition, or used frozen as features for phoneme recognition, keyword spotting, intent and emotion.
- **Textless NLP:** Generative Spoken Language Modeling (Lakhotia et al., 2021) trains a language model on deduplicated HuBERT units and resynthesises speech from them, with no text anywhere.
- **Speech-to-speech and resynthesis:** Discrete units serve as compact targets for speech-to-unit translation and unit-to-waveform vocoders.
- **Teacher for later models:** **WavLM** keeps HuBERT's masked unit prediction and adds denoising; DistilHuBERT compresses it by layer-wise distillation.

> **Units are not phones:** k-means units over-segment: one phone usually spans several unit ids, and a unit can change in the middle of a steady vowel. They also carry speaker and channel information. Collapse repeats before language modelling, and pick the layer and $k$ by PNMI or downstream results, not by habit. Different layers give very different units.

**Question:** Why does HuBERT compute the loss on masked frames only (α = 1)?

- A. Unmasked frames have no k-means label
- B. Predicting a visible frame's label only reproduces the clustering, errors included; hidden frames force the model to use context, which smooths over noisy targets
- C. It halves the compute
- D. The softmax is undefined for unmasked frames

**Answer:** B. Predicting a visible frame's label only reproduces the clustering, errors included; hidden frames force the model to use context, which smooths over noisy targets — With the frame visible, the easiest solution is to imitate k-means on the local features. Hiding it makes the model infer the unit from surrounding speech: acoustic and language structure that also helps downstream tasks. The paper found α = 1 works best, especially when targets are poor.

**Question:** Going from k = 100 to k = 500 clusters on the same features usually…

- A. raises both phone purity and cluster purity
- B. raises phone purity and lowers cluster purity, so PNMI is the fairer comparison
- C. lowers both
- D. leaves PNMI unchanged by definition

**Answer:** B. raises phone purity and lowers cluster purity, so PNMI is the fairer comparison — Smaller clusters are purer, but each phone is split over more of them. PNMI = I(y; z)/H(y) measures how much phone information the unit carries, regardless of how finely it is split.

**Key points**

- Targets are offline k-means labels: MFCCs with k = 100, then layer-6 features of the iteration-1 model with k = 500.
- Loss: cross-entropy over units, $\cos(A o_t,e_c)/\tau$ with $\tau=0.1$, on masked frames only ($\alpha=1$).
- Masking as in wav2vec 2.0 (p = 8%, M = 10). No distractors, diversity loss or Gumbel schedule.
- Iterating improves the targets: PNMI of learned features is far higher than MFCCs.
- BASE 95M, LARGE 317M, X-LARGE 964M. Matches or beats wav2vec 2.0. Its units power textless speech LMs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
