Definition
WavLM is a self-supervised speech encoder trained by masked speech denoising and prediction. Its input is an utterance artificially overlapped with another utterance or with noise. On masked frames it predicts the discrete pseudo-labels of the original utterance, HuBERT-style. Introduced by Chen et al. (Microsoft), "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing" (IEEE Journal of Selected Topics in Signal Processing, 2022).
wav2vec 2.0 and HuBERT were built for recognition. They are pre-trained on clean, single-speaker audiobooks, and their objectives reward knowing what was said. Many speech tasks need more. Speaker verification and Speaker diarization need who is speaking, separation needs which voice is which, and real recordings contain overlap and noise. WavLM's goal was one encoder for the "full stack": content, speaker, paralinguistics and multi-talker audio.
It keeps HuBERT's recipe (CNN encoder, span masking, cross-entropy on k-means units at masked frames) and changes three things: the input is corrupted by utterance mixing, the Transformer uses a gated relative position bias, and the data grows to 94k hours from more varied sources.
Masked speech denoising
For some utterances in a batch, a random chunk of another utterance, or of noise, is added at a random position with a random energy ratio. The chunk is shorter than half the primary utterance, so the primary speaker is always the majority and "whose content to predict" is well defined. The targets are still the units of the clean primary. On a masked frame inside the overlap, the model hears two voices, sees neither directly, and must output the primary speaker's unit. To do that it has to track which voice is which, and that is speaker information.
- learned scalar bias for the (bucketed) relative distance , as in T5
- learned vectors; the gates depend on the query , so each frame can decide how much distance matters
- learned scalar
- attention weight from frame to frame (see Self-attention)
wav2vec 2.0 and HuBERT add position through a convolution on the input. A relative bias acts inside every attention score, and the gates let the content of the current frame scale it. For example, a silent or noisy frame can pay less attention to near neighbours. The paper reports that the gated bias improved results over a plain relative bias.
Models and data (Chen et al., 2022)
- BASE
- 12 layers, width 768, about 95M parameters; LibriSpeech 960 h
- BASE+
- same size as BASE; 94k h
- LARGE
- 24 layers, width 1024, about 317M parameters; 94k h
- 94k hours
- Libri-Light 60k h (audiobooks) + GigaSpeech 10k h (podcasts, YouTube) + VoxPopuli 24k h (European Parliament), all English
- Targets
- k-means pseudo-labels in the HuBERT manner, from an earlier model's features
- SUPERB
- WavLM LARGE ranked first on the SUPERB benchmark at publication; the largest gains were on speaker verification, diarization and separation
Worked example: setting the mix level
Primary 4 s; interferer chunk of 1.2 s (30% overlap) at a 5 dB energy ratio
- 1Primary energy over the whole utterance , inside the chosen region . Energy of the raw interferer chunk .
- 2Gain for a ratio of dB inside the region: .
- 3Scaled interferer energy: , so the ratio inside the region is dB, as intended.
- 4Over the whole utterance: dB, which equals . A short overlap looks mild in utterance SNR but is just as hard locally.
- 5Training pair: input = primary + chunk; targets = units of the primary alone, used on masked frames only.
1import numpy as np23def mix_utterance(primary, other, rng, max_ratio=0.5, snr_db=(-5.0, 5.0)):4 """WavLM-style utterance mixing: add a random chunk of `other`, shorter than5 max_ratio of the primary, at a random position and energy ratio (dB)."""6 n = len(primary)7 length = rng.integers(1, int(max_ratio * n)) # < half: primary stays the majority8 start = rng.integers(0, n - length + 1)9 src = rng.integers(0, len(other) - length + 1)10 chunk = other[src:src + length]11 region = primary[start:start + length]12 ratio = rng.uniform(*snr_db)13 gain = np.sqrt(np.sum(region**2) / (np.sum(chunk**2) * 10 ** (ratio / 10)))14 mixed = primary.copy()15 mixed[start:start + length] += gain * chunk16 return mixed, (start, length, ratio)1718sr = 16_00019t = np.arange(4 * sr) / sr20rng = np.random.default_rng(0)21primary = np.sin(2 * np.pi * 140 * t) * (0.6 + 0.4 * np.sin(2 * np.pi * 3 * t))22other = np.sin(2 * np.pi * 230 * t) * (0.6 + 0.4 * np.cos(2 * np.pi * 2 * t))23mixed, (start, length, ratio) = mix_utterance(primary, other, rng)24interf = mixed - primary25print(f"overlap {length / len(primary):.1%} at {start / sr:.2f} s, ratio {ratio:.2f} dB in the region,",26 f"{10 * np.log10(np.sum(primary**2) / np.sum(interf**2)):.2f} dB over the utterance")27# The model gets `mixed`; its targets are the k-means units of `primary`.Where it sits
| Aspect | wav2vec 2.0 | HuBERT | WavLM |
|---|---|---|---|
| Input | clean speech | clean speech | speech mixed with speech or noise |
| Targets | jointly learned quantised latents | offline k-means units | offline k-means units of the clean primary |
| Loss | contrastive, 100 distractors | cross-entropy, masked frames | cross-entropy, masked frames |
| Position | conv embedding | conv embedding | conv embedding + gated relative bias |
| Largest data | LV-60k (53.2k h) | Libri-Light 60k h | 94k h, three sources |
In practice WavLM is used like HuBERT: load a checkpoint such as microsoft/wavlm-base-plus or microsoft/wavlm-large in Hugging Face transformers and read the hidden states. Speaker verification adds an x-vector or ECAPA-style head on top, and diarization pipelines use it to embed short windows.
Use when
- +One encoder for content, speaker, emotion and multi-talker tasks; strong on SUPERB
- +Robust to overlap and noise by construction
- +Same interface as HuBERT: drop-in for existing pipelines
Avoid when
- −Needs an earlier model to produce pseudo-labels, plus a mixing and noise pipeline
- −English-only pre-training data; multilingual use needs other checkpoints
- −ASR gains over HuBERT are small; the extra cost pays off mainly for speaker and overlap tasks
Check yourself
In WavLM's denoising task, what are the targets for a masked frame inside the overlapped region?
Check yourself
Why is the interfering chunk kept shorter than half the primary utterance?
Remember
- ✓WavLM = HuBERT-style masked unit prediction + denoising: the input is mixed with another utterance or noise, the targets come from the clean primary.
- ✓The interfering chunk is shorter than half the utterance, so the primary speaker is the majority.
- ✓Gated relative position bias: is added to every attention score, and query-dependent gates scale it.
- ✓BASE 95M on LibriSpeech 960 h; BASE+ and LARGE (317M) on 94k h: Libri-Light 60k + GigaSpeech 10k + VoxPopuli 24k.
- ✓Led SUPERB; biggest gains on speaker verification, diarization and separation.