Definition
HuBERT (Hidden-Unit BERT) is a self-supervised speech encoder trained by masked prediction of offline cluster labels. k-means assigns every 20 ms frame a discrete "hidden unit". The model sees the waveform with spans masked and must classify the unit of each masked frame. Introduced by Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov and Mohamed, "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units" (Meta AI, IEEE/ACM TASLP 2021).
The input is raw 16 kHz audio and the output is one vector per 20 ms frame. Speech has no given vocabulary of tokens: unlike text, there is no lexicon to mask and predict, and segment boundaries are unknown. wav2vec 2.0 solved this by learning a quantiser jointly with the encoder and using a contrastive loss. That needs distractor sampling, a diversity loss and a Gumbel temperature schedule, and the codebook can still collapse.
HuBERT splits the problem in two. Targets are produced offline by plain k-Means, with no gradients. The model is then trained with an ordinary cross-entropy loss, as in BERT. The targets need not be good. They only need to be consistent, because the model learns from the masked frames' context, and its own features can then be re-clustered into better targets.
Targets and model
Iteration 1. k-means with on 39-dimensional MFCC features (13 coefficients plus first and second deltas). Iteration 2. k-means with on the output of layer 6 of the iteration-1 model. The model itself uses the CNN feature encoder of wav2vec 2.0 (stride 320, 20 ms) and a Transformer. Spans are masked as in wav2vec 2.0, with of frames chosen as starts and span length .
- the input with frames in the masked set replaced by a learned mask embedding
- Transformer output at frame and a learned projection
- learned embedding of unit , one of clusters;
- k-means label of frame
The full loss is , where is the same cross-entropy on unmasked frames. The paper found works best: loss on masked frames only. Predicting a visible frame's unit just copies the clustering, errors included. Predicting a hidden frame's unit forces the model to infer it from context, and context averages out the clustering's noise. This is also why noisy MFCC targets work at all.
Measuring target quality
The paper scores clusterings against forced-aligned phone labels , with three numbers. Phone purity : how pure each cluster is. Cluster purity : how concentrated each phone is. PNMI : the share of phone uncertainty that the unit explains. More clusters raise phone purity and lower cluster purity, so PNMI is the fairer single number. Features from the iteration-1 model give much higher PNMI than MFCCs, which is the point of iterating.
Configurations (Hsu et al., 2021)
- BASE
- 12 layers, width 768; about 95M parameters; 2 iterations on LibriSpeech 960 h
- LARGE
- 24 layers, width 1024; about 317M parameters; Libri-Light 60k h
- X-LARGE
- 48 layers, width 1280; about 964M parameters; Libri-Light 60k h
- Targets for LARGE / X-LARGE
- one iteration, k = 500 on layer 9 of the second-iteration BASE model
- Results
- matches or beats wav2vec 2.0 on LibriSpeech and Libri-light fine-tuning splits (10 min to 960 h); X-LARGE gives up to 19% and 13% relative WER reduction on dev-other and test-other
Compared with wav2vec 2.0
| Aspect | wav2vec 2.0 | HuBERT |
|---|---|---|
| Targets | quantised latents, learned jointly (2 × 320 codebooks) | k-means ids, computed offline (k = 100, then 500) |
| Loss | InfoNCE: target vs 100 distractors | cross-entropy over all C units |
| Extra terms | diversity loss, Gumbel temperature schedule | none |
| Masking | p = 0.065, M = 10 | p = 8%, M = 10 |
| Training | one stage | iterative: train, re-cluster, retrain |
The price of the simpler loss is the extra stages: clustering hundreds of hours of features and retraining from scratch. The gain is stability. The targets are fixed during a training run, so the model cannot make its task easier by changing them, as a jointly learned quantiser can.
Worked example: purity and PNMI
Ten frames with phones = a a a a b b b b b b and units = 1 1 1 2 2 2 3 3 3 3.
| Quantity | Computation | Value |
|---|---|---|
| joint counts | (a,1) = 3, (a,2) = 1, (b,2) = 2, (b,3) = 4 | 10 frames |
| marginals | p(a) = 0.4, p(b) = 0.6 · p(z) = 0.3, 0.3, 0.4 | |
| phone purity | (3 + 2 + 4) / 10 (majority phone of each unit) | 0.900 |
| cluster purity | (3 + 4) / 10 (majority unit of each phone) | 0.700 |
| H(y) | −0.4 log 0.4 − 0.6 log 0.6 | 0.971 bits |
| I(y; z) | 0.3 log(0.3/0.12) + 0.1 log(0.1/0.12) + 0.2 log(0.2/0.18) + 0.4 log(0.4/0.24) | 0.695 bits |
| PNMI | 0.695 / 0.971 | 0.716 |
Unit 2 is mixed. It covers the end of a and the start of b, which is typical at phone boundaries. Collapsing repeats gives the unit string 1 2 3, the form that unit language models are trained on. For the prediction itself, suppose is 0.7, 0.4 and −0.2 for three units and the target is the first. At the softmax gives it 0.953, a loss of 0.049 nats.
1import numpy as np, soundfile as sf, torch2from sklearn.cluster import MiniBatchKMeans3from transformers import AutoFeatureExtractor, HubertModel45fe = AutoFeatureExtractor.from_pretrained("facebook/hubert-base-ls960")6model = HubertModel.from_pretrained("facebook/hubert-base-ls960").eval()78def layer_features(path, layer=6):9 wav, sr = sf.read(path, dtype="float32") # mono, 16 kHz10 inputs = fe(wav, sampling_rate=sr, return_tensors="pt")11 with torch.no_grad():12 out = model(**inputs, output_hidden_states=True)13 return out.hidden_states[layer][0].numpy() # (frames, 768), 50 per second1415feats = np.concatenate([layer_features(p) for p in train_files])16km = MiniBatchKMeans(n_clusters=500, batch_size=10_000, n_init=3).fit(feats)1718units = km.predict(layer_features("utterance.wav")) # one unit id per 20 ms frame19dedup = units[np.r_[True, units[1:] != units[:-1]]] # collapse repeats for unit LMsWhere it's used
ASR and SUPERB tasks
Fine-tuned with Connectionist Temporal Classification for recognition, or used frozen as features for phoneme recognition, keyword spotting, intent and emotion.
Textless NLP
Generative Spoken Language Modeling (Lakhotia et al., 2021) trains a language model on deduplicated HuBERT units and resynthesises speech from them, with no text anywhere.
Speech-to-speech and resynthesis
Discrete units serve as compact targets for speech-to-unit translation and unit-to-waveform vocoders.
Teacher for later models
WavLM keeps HuBERT's masked unit prediction and adds denoising; DistilHuBERT compresses it by layer-wise distillation.
Check yourself
Why does HuBERT compute the loss on masked frames only (α = 1)?
Check yourself
Going from k = 100 to k = 500 clusters on the same features usually…
Remember
- ✓Targets are offline k-means labels: MFCCs with k = 100, then layer-6 features of the iteration-1 model with k = 500.
- ✓Loss: cross-entropy over units, with , on masked frames only ().
- ✓Masking as in wav2vec 2.0 (p = 8%, M = 10). No distractors, diversity loss or Gumbel schedule.
- ✓Iterating improves the targets: PNMI of learned features is far higher than MFCCs.
- ✓BASE 95M, LARGE 317M, X-LARGE 964M. Matches or beats wav2vec 2.0. Its units power textless speech LMs.