HuBERT

HuBERT clusters frames offline with k-means (MFCCs, k = 100, then its own layer features, k = 500) and trains a Transformer to predict the cluster id of masked frames. Each round of re-clustering gives better targets.

8 min read711 wordsadvanced0 viewsupdated 2026-10-15

Definition

HuBERT (Hidden-Unit BERT) is a self-supervised speech encoder trained by masked prediction of offline cluster labels. k-means assigns every 20 ms frame a discrete "hidden unit". The model sees the waveform with spans masked and must classify the unit of each masked frame. Introduced by Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov and Mohamed, "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units" (Meta AI, IEEE/ACM TASLP 2021).

The input is raw 16 kHz audio and the output is one vector per 20 ms frame. Speech has no given vocabulary of tokens: unlike text, there is no lexicon to mask and predict, and segment boundaries are unknown. wav2vec 2.0 solved this by learning a quantiser jointly with the encoder and using a contrastive loss. That needs distractor sampling, a diversity loss and a Gumbel temperature schedule, and the codebook can still collapse.

HuBERT splits the problem in two. Targets are produced offline by plain k-Means, with no gradients. The model is then trained with an ordinary cross-entropy loss, as in BERT. The targets need not be good. They only need to be consistent, because the model learns from the masked frames' context, and its own features can then be re-clustered into better targets.

Targets and model

Iteration 1. k-means with k=100k=100 on 39-dimensional MFCC features (13 coefficients plus first and second deltas). Iteration 2. k-means with k=500k=500 on the output of layer 6 of the iteration-1 model. The model itself uses the CNN feature encoder of wav2vec 2.0 (stride 320, 20 ms) and a Transformer. Spans are masked as in wav2vec 2.0, with p=8%p=8\% of frames chosen as starts and span length M=10M=10.

offline target generation (no gradients)featuresiter 1: 39-d MFCC(13 + Δ + ΔΔ)iter 2: layer 6k-meansiter 1: k = 100iter 2: k = 500one unit/frametargets z_t (cluster ids)171744488888831316262used only where the frame is maskedBASE: 2 iterations on LibriSpeech 960 h (iter 2 clusters layer 6 of the iter-1 model)LARGE / X-LARGE: 1 iteration on Libri-Light 60k h, units from BASE iter-2 layer 9, k = 500waveform 16 kHzCNN encoderas wav2vec 2.0stride 320 = 20 msmask spans: p = 8% starts, M = 10x1x2MMMMMMMx10x11x12Transformer → o_tBASE 12 × 768 · LARGE 24 × 1024X-LARGE 48 × 1280––p4p4p4p88p88p88p31–––next round:re-cluster amiddle layerof this modelp(c | x̃, t) = softmax over the C units of cos(A·o_t, e_c) / τ, τ = 0.1A: projection · e_c: learned embedding of unit closs = cross-entropy against z_t on masked frames only (α = 1)unmasked frames carry no loss: the model must infer hidden units from context
fig.HuBERT training. Top: offline target generation. Frame features (MFCCs in iteration 1, a middle Transformer layer later) are clustered by k-means into one unit id per frame. Bottom: the model reads the waveform through the CNN encoder, spans are masked (p = 8%, M = 10) and the Transformer predicts each masked frame's unit with a softmax over learned unit embeddings. Unmasked frames carry no loss. After training, a middle layer is re-clustered for the next iteration (dashed arrow).
p(c∣X~,t)=exp⁡ ⁣(cos⁡(A ot, ec)/τ)∑c′=1Cexp⁡ ⁣(cos⁡(A ot, ec′)/τ),Lm=−∑t∈Mlog⁡p(zt∣X~,t)p(c\mid\tilde X,t)=\frac{\exp\!\big(\cos(A\,o_t,\,e_c)/\tau\big)}{\sum_{c'=1}^{C}\exp\!\big(\cos(A\,o_t,\,e_{c'})/\tau\big)},\qquad \mathcal L_m=-\sum_{t\in\mathcal M}\log p(z_t\mid\tilde X,t)
X~\tilde X
the input with frames in the masked set M\mathcal M replaced by a learned mask embedding
ot,  Ao_t,\;A
Transformer output at frame tt and a learned projection
ece_c
learned embedding of unit cc, one of CC clusters; τ=0.1\tau=0.1
ztz_t
k-means label of frame tt

The full loss is α Lm+(1−α) Lu\alpha\,\mathcal L_m+(1-\alpha)\,\mathcal L_u, where Lu\mathcal L_u is the same cross-entropy on unmasked frames. The paper found α=1\alpha=1 works best: loss on masked frames only. Predicting a visible frame's unit just copies the clustering, errors included. Predicting a hidden frame's unit forces the model to infer it from context, and context averages out the clustering's noise. This is also why noisy MFCC targets work at all.

Measuring target quality

The paper scores clusterings against forced-aligned phone labels yy, with three numbers. Phone purity Ez[max⁡yp(y∣z)]\mathbb E_z[\max_y p(y\mid z)]: how pure each cluster is. Cluster purity Ey[max⁡zp(z∣y)]\mathbb E_y[\max_z p(z\mid y)]: how concentrated each phone is. PNMI =I(y;z)/H(y)=I(y;z)/H(y): the share of phone uncertainty that the unit explains. More clusters raise phone purity and lower cluster purity, so PNMI is the fairer single number. Features from the iteration-1 model give much higher PNMI than MFCCs, which is the point of iterating.

600 frames · colour = phone · + = centroiditer 1: per-frame (MFCC-like)target quality vs k00.512346812162432solid PNMI · dashed phone purity · iter 1 · iter 2iteration 1, k = 6phone purity 0.638 · cluster purity 0.638PNMI = I(phone; unit) / H(phone) = 0.442iteration 2 at this k: PNMI 0.553frames 120–159 (800 ms): true phone (top) and HuBERT target unit (bottom)4442110000311310525525255335242222412444runs collapsed: 5 phone segments → 25 units (shaded cells start a new unit)dedup units: 4 2 1 0 3 1 3 1 0 5 2 5 2 5 2 5 3 5 2 4 2 4 1 2 4toy: 6 phones, 2-d features, noise σ = 0.45 · the loss is cross-entropy on these units at masked frames
targets from
feature noise σ
fig.k-means targets on a toy 600-frame utterance with 6 phones and noisy 2-d features (σ = 0.45). At the default k = 6, iteration 1 (per-frame, MFCC-like features) has phone purity 0.638, cluster purity 0.638 and PNMI 0.442. Iteration 2 averages each frame with its ±2 neighbours, a stand-in for the context a trained model uses, and PNMI rises to 0.553. The unit strip shows over-segmentation: 5 phone segments become 25 units in iteration 1 but 12 in iteration 2. At k = 32, phone purity is 0.622 versus 0.788 and PNMI 0.528 versus 0.714, while cluster purity falls to 0.208 and 0.260. At k = 2, cluster purity is 0.860, yet PNMI is only 0.189.

Configurations (Hsu et al., 2021)

BASE
12 layers, width 768; about 95M parameters; 2 iterations on LibriSpeech 960 h
LARGE
24 layers, width 1024; about 317M parameters; Libri-Light 60k h
X-LARGE
48 layers, width 1280; about 964M parameters; Libri-Light 60k h
Targets for LARGE / X-LARGE
one iteration, k = 500 on layer 9 of the second-iteration BASE model
Results
matches or beats wav2vec 2.0 on LibriSpeech and Libri-light fine-tuning splits (10 min to 960 h); X-LARGE gives up to 19% and 13% relative WER reduction on dev-other and test-other

Compared with wav2vec 2.0

Aspectwav2vec 2.0HuBERT
Targetsquantised latents, learned jointly (2 × 320 codebooks)k-means ids, computed offline (k = 100, then 500)
LossInfoNCE: target vs 100 distractorscross-entropy over all C units
Extra termsdiversity loss, Gumbel temperature schedulenone
Maskingp = 0.065, M = 10p = 8%, M = 10
Trainingone stageiterative: train, re-cluster, retrain

The price of the simpler loss is the extra stages: clustering hundreds of hours of features and retraining from scratch. The gain is stability. The targets are fixed during a training run, so the model cannot make its task easier by changing them, as a jointly learned quantiser can.

Worked example: purity and PNMI

Ten frames with phones yy = a a a a b b b b b b and units zz = 1 1 1 2 2 2 3 3 3 3.

QuantityComputationValue
joint counts(a,1) = 3, (a,2) = 1, (b,2) = 2, (b,3) = 410 frames
marginalsp(a) = 0.4, p(b) = 0.6 · p(z) = 0.3, 0.3, 0.4
phone purity(3 + 2 + 4) / 10 (majority phone of each unit)0.900
cluster purity(3 + 4) / 10 (majority unit of each phone)0.700
H(y)−0.4 log 0.4 − 0.6 log 0.60.971 bits
I(y; z)0.3 log(0.3/0.12) + 0.1 log(0.1/0.12) + 0.2 log(0.2/0.18) + 0.4 log(0.4/0.24)0.695 bits
PNMI0.695 / 0.9710.716
Counts and metrics, logs base 2. Computed in Python.

Unit 2 is mixed. It covers the end of a and the start of b, which is typical at phone boundaries. Collapsing repeats gives the unit string 1 2 3, the form that unit language models are trained on. For the prediction itself, suppose cos⁡(Aot,ec)\cos(A o_t,e_c) is 0.7, 0.4 and −0.2 for three units and the target is the first. At τ=0.1\tau=0.1 the softmax gives it 0.953, a loss of 0.049 nats.

python
1import numpy as np, soundfile as sf, torch
2from sklearn.cluster import MiniBatchKMeans
3from transformers import AutoFeatureExtractor, HubertModel
4
5fe = AutoFeatureExtractor.from_pretrained("facebook/hubert-base-ls960")
6model = HubertModel.from_pretrained("facebook/hubert-base-ls960").eval()
7
8def layer_features(path, layer=6):
9 wav, sr = sf.read(path, dtype="float32") # mono, 16 kHz
10 inputs = fe(wav, sampling_rate=sr, return_tensors="pt")
11 with torch.no_grad():
12 out = model(**inputs, output_hidden_states=True)
13 return out.hidden_states[layer][0].numpy() # (frames, 768), 50 per second
14
15feats = np.concatenate([layer_features(p) for p in train_files])
16km = MiniBatchKMeans(n_clusters=500, batch_size=10_000, n_init=3).fit(feats)
17
18units = km.predict(layer_features("utterance.wav")) # one unit id per 20 ms frame
19dedup = units[np.r_[True, units[1:] != units[:-1]]] # collapse repeats for unit LMs
Extract HuBERT units with Hugging Face transformers and scikit-learn: a middle layer, k-means with 500 clusters, one id per 20 ms. Not run here (no torch/transformers in this environment). train_files is your list of 16 kHz mono WAVs.

Where it's used

ASR and SUPERB tasks

Fine-tuned with Connectionist Temporal Classification for recognition, or used frozen as features for phoneme recognition, keyword spotting, intent and emotion.

Textless NLP

Generative Spoken Language Modeling (Lakhotia et al., 2021) trains a language model on deduplicated HuBERT units and resynthesises speech from them, with no text anywhere.

Speech-to-speech and resynthesis

Discrete units serve as compact targets for speech-to-unit translation and unit-to-waveform vocoders.

Teacher for later models

WavLM keeps HuBERT's masked unit prediction and adds denoising; DistilHuBERT compresses it by layer-wise distillation.

Check yourself

Why does HuBERT compute the loss on masked frames only (α = 1)?

Check yourself

Going from k = 100 to k = 500 clusters on the same features usually…

Remember

  • ✓Targets are offline k-means labels: MFCCs with k = 100, then layer-6 features of the iteration-1 model with k = 500.
  • ✓Loss: cross-entropy over units, cos⁡(Aot,ec)/τ\cos(A o_t,e_c)/\tau with τ=0.1\tau=0.1, on masked frames only (α=1\alpha=1).
  • ✓Masking as in wav2vec 2.0 (p = 8%, M = 10). No distractors, diversity loss or Gumbel schedule.
  • ✓Iterating improves the targets: PNMI of learned features is far higher than MFCCs.
  • ✓BASE 95M, LARGE 317M, X-LARGE 964M. Matches or beats wav2vec 2.0. Its units power textless speech LMs.
← back to Deep Learning roadmap