WavLM

WavLM is HuBERT plus denoising. It hears a primary utterance mixed with another speaker or noise, and predicts the primary's k-means units at masked frames. With gated relative position bias and 94k h of data it led SUPERB.

7 min read665 wordsadvanced0 viewsupdated 2026-10-15

Definition

WavLM is a self-supervised speech encoder trained by masked speech denoising and prediction. Its input is an utterance artificially overlapped with another utterance or with noise. On masked frames it predicts the discrete pseudo-labels of the original utterance, HuBERT-style. Introduced by Chen et al. (Microsoft), "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing" (IEEE Journal of Selected Topics in Signal Processing, 2022).

wav2vec 2.0 and HuBERT were built for recognition. They are pre-trained on clean, single-speaker audiobooks, and their objectives reward knowing what was said. Many speech tasks need more. Speaker verification and Speaker diarization need who is speaking, separation needs which voice is which, and real recordings contain overlap and noise. WavLM's goal was one encoder for the "full stack": content, speaker, paralinguistics and multi-talker audio.

It keeps HuBERT's recipe (CNN encoder, span masking, cross-entropy on k-means units at masked frames) and changes three things: the input is corrupted by utterance mixing, the Transformer uses a gated relative position bias, and the data grows to 94k hours from more varied sources.

Masked speech denoising

For some utterances in a batch, a random chunk of another utterance, or of noise, is added at a random position with a random energy ratio. The chunk is shorter than half the primary utterance, so the primary speaker is always the majority and "whose content to predict" is well defined. The targets are still the units of the clean primary. On a masked frame inside the overlap, the model hears two voices, sees neither directly, and must output the primary speaker's unit. To do that it has to track which voice is which, and that is speaker information.

masked speech denoising and prediction · Chen et al. (2022)primary uother speech / noiseutterance mixingcrop a random chunkshorter than half of urandom positionrandom energy ratioinput x = u + interfereroverlap < 50%CNN encoder20 ms framesx1x2MMMMx7x8x9MMMMx14Transformer + gated relative position biasbias on q·k depends on distance i − j, gated by the query––ŷŷŷŷ–––ŷŷŷŷ–targets z_tk-means units of theCLEAN primary uloss: cross-entropy of ŷ_t against z_t on masked frames (HuBERT-style)masked frames inside the overlap (orange outline) force denoising: the interferer isaudible, yet the target is the primary speaker's content → speaker-aware featuresWavLM Large: 94k h = Libri-Light 60k + GigaSpeech 10k + VoxPopuli 24k (English)BASE: 12 × 768 on LibriSpeech 960 h · BASE+: 12 × 768 on 94k h · LARGE: 24 × 1024
fig.WavLM pre-training. A primary utterance u is mixed with a random chunk of another utterance or noise, shorter than half of u. The mixture passes through the CNN encoder (20 ms frames), span masking and a Transformer with gated relative position bias. On masked frames, the output is trained with cross-entropy against the k-means units of the clean u. Masked frames inside the overlap (orange outline) are where denoising is learned. BASE is trained on LibriSpeech 960 h; BASE+ and LARGE on 94k h.
giupd,  girst=σ(qi ⁣⋅u),  σ(qi ⁣⋅w),ri−j=di−j+giupd di−j+(1−giupd) s girst di−j,aij∝exp⁡ ⁣(qi⋅kjdk+ri−j)g^{\text{upd}}_i,\;g^{\text{rst}}_i=\sigma(q_i\!\cdot u),\;\sigma(q_i\!\cdot w),\qquad r_{i-j}=d_{i-j}+g^{\text{upd}}_i\,d_{i-j}+\big(1-g^{\text{upd}}_i\big)\,s\,g^{\text{rst}}_i\,d_{i-j},\qquad a_{ij}\propto\exp\!\Big(\frac{q_i\cdot k_j}{\sqrt{d_k}}+r_{i-j}\Big)
di−jd_{i-j}
learned scalar bias for the (bucketed) relative distance i−ji-j, as in T5
u, wu,\,w
learned vectors; the gates depend on the query qiq_i, so each frame can decide how much distance matters
ss
learned scalar
aija_{ij}
attention weight from frame ii to frame jj (see Self-attention)

wav2vec 2.0 and HuBERT add position through a convolution on the input. A relative bias acts inside every attention score, and the gates let the content of the current frame scale it. For example, a silent or noisy frame can pay less attention to near neighbours. The paper reports that the gated bias improved results over a plain relative bias.

Models and data (Chen et al., 2022)

BASE
12 layers, width 768, about 95M parameters; LibriSpeech 960 h
BASE+
same size as BASE; 94k h
LARGE
24 layers, width 1024, about 317M parameters; 94k h
94k hours
Libri-Light 60k h (audiobooks) + GigaSpeech 10k h (podcasts, YouTube) + VoxPopuli 24k h (European Parliament), all English
Targets
k-means pseudo-labels in the HuBERT manner, from an earlier model's features
SUPERB
WavLM LARGE ranked first on the SUPERB benchmark at publication; the largest gains were on speaker verification, diarization and separation
2 s utterance, 100 frames of 20 ms · interferer over 30% of it at 5 dBcleanvoice 2mixturemasktarget0 s0.5 s1 s1.5 s2 s■ masked, unit of mix = unit of clean · ■ masked, unit would change if taken from the mixoverlap frames 57–86 (30/100) · energy ratio in overlap 5.00 dB · whole utterance 11.27 dBmasked frames 53/100 (p = 0.08, M = 10) · inside the overlap 11masked overlap frames whose target would change if computed from the mix: 4/11all overlap frames with a changed unit: 10/30 · interferer gain 0.657under 50% overlap the primary speaker stays the majority: its units are the targetstoy: 2 kHz sample rate · unit = strongest 50 Hz DFT bin of the frame, or silencethe model hears the mixture but is scored only on masked frames, against the clean units
interferer#1
fig.Utterance mixing on a toy 2 s utterance (100 frames of 20 ms; toy 2 kHz sample rate). Defaults: a second voice covers 30% of it (frames 57–86) at an energy ratio of 5 dB inside the overlap. Over the whole utterance this is 11.27 dB, because the interferer is silent elsewhere. Span masking (p = 0.08, M = 10) hides 53 frames, 11 of them inside the overlap. For 4 of those 11, the toy unit of the mixture differs from the clean unit, and 10 of the 30 overlap frames change overall. Those frames would get wrong targets if units were computed from the mix, which is why WavLM labels the clean primary. At 20 dB only 1 masked frame (2 of 30 overall) changes, and the utterance SNR is 26.27 dB. With noise instead of a voice at 5 dB, it is 5 of 11. Above 50% overlap the figure warns that the main speaker becomes ambiguous.

Worked example: setting the mix level

Primary 4 s; interferer chunk of 1.2 s (30% overlap) at a 5 dB energy ratio

  1. 1Primary energy over the whole utterance Ep=100E_p=100, inside the chosen region Epreg=30E_p^{\text{reg}}=30. Energy of the raw interferer chunk Es=50E_s=50.
  2. 2Gain for a ratio of r=5r=5 dB inside the region: g=Epreg/(Es⋅10r/10)=30/(50⋅3.162)=0.436g=\sqrt{E_p^{\text{reg}}/(E_s\cdot10^{r/10})}=\sqrt{30/(50\cdot3.162)}=0.436.
  3. 3Scaled interferer energy: g2Es=0.190⋅50=9.49g^2E_s=0.190\cdot50=9.49, so the ratio inside the region is 10log⁡10(30/9.49)=510\log_{10}(30/9.49)=5 dB, as intended.
  4. 4Over the whole utterance: 10log⁡10(100/9.49)=10.2310\log_{10}(100/9.49)=10.23 dB, which equals 5+10log⁡10(100/30)5+10\log_{10}(100/30). A short overlap looks mild in utterance SNR but is just as hard locally.
  5. 5Training pair: input = primary + 0.436⋅0.436\cdotchunk; targets = units of the primary alone, used on masked frames only.
python
1import numpy as np
2
3def mix_utterance(primary, other, rng, max_ratio=0.5, snr_db=(-5.0, 5.0)):
4 """WavLM-style utterance mixing: add a random chunk of `other`, shorter than
5 max_ratio of the primary, at a random position and energy ratio (dB)."""
6 n = len(primary)
7 length = rng.integers(1, int(max_ratio * n)) # < half: primary stays the majority
8 start = rng.integers(0, n - length + 1)
9 src = rng.integers(0, len(other) - length + 1)
10 chunk = other[src:src + length]
11 region = primary[start:start + length]
12 ratio = rng.uniform(*snr_db)
13 gain = np.sqrt(np.sum(region**2) / (np.sum(chunk**2) * 10 ** (ratio / 10)))
14 mixed = primary.copy()
15 mixed[start:start + length] += gain * chunk
16 return mixed, (start, length, ratio)
17
18sr = 16_000
19t = np.arange(4 * sr) / sr
20rng = np.random.default_rng(0)
21primary = np.sin(2 * np.pi * 140 * t) * (0.6 + 0.4 * np.sin(2 * np.pi * 3 * t))
22other = np.sin(2 * np.pi * 230 * t) * (0.6 + 0.4 * np.cos(2 * np.pi * 2 * t))
23mixed, (start, length, ratio) = mix_utterance(primary, other, rng)
24interf = mixed - primary
25print(f"overlap {length / len(primary):.1%} at {start / sr:.2f} s, ratio {ratio:.2f} dB in the region,",
26 f"{10 * np.log10(np.sum(primary**2) / np.sum(interf**2)):.2f} dB over the utterance")
27# The model gets `mixed`; its targets are the k-means units of `primary`.
WavLM-style utterance mixing in numpy. The 50% cap follows the paper; the −5…5 dB range is illustrative. Prints: overlap 42.5% at 1.46 s, ratio −4.59 dB in the region, −0.90 dB over the utterance (run here).

Where it sits

Aspectwav2vec 2.0HuBERTWavLM
Inputclean speechclean speechspeech mixed with speech or noise
Targetsjointly learned quantised latentsoffline k-means unitsoffline k-means units of the clean primary
Losscontrastive, 100 distractorscross-entropy, masked framescross-entropy, masked frames
Positionconv embeddingconv embeddingconv embedding + gated relative bias
Largest dataLV-60k (53.2k h)Libri-Light 60k h94k h, three sources

In practice WavLM is used like HuBERT: load a checkpoint such as microsoft/wavlm-base-plus or microsoft/wavlm-large in Hugging Face transformers and read the hidden states. Speaker verification adds an x-vector or ECAPA-style head on top, and diarization pipelines use it to embed short windows.

Use when

  • +One encoder for content, speaker, emotion and multi-talker tasks; strong on SUPERB
  • +Robust to overlap and noise by construction
  • +Same interface as HuBERT: drop-in for existing pipelines

Avoid when

  • −Needs an earlier model to produce pseudo-labels, plus a mixing and noise pipeline
  • −English-only pre-training data; multilingual use needs other checkpoints
  • −ASR gains over HuBERT are small; the extra cost pays off mainly for speaker and overlap tasks

Check yourself

In WavLM's denoising task, what are the targets for a masked frame inside the overlapped region?

Check yourself

Why is the interfering chunk kept shorter than half the primary utterance?

Remember

  • ✓WavLM = HuBERT-style masked unit prediction + denoising: the input is mixed with another utterance or noise, the targets come from the clean primary.
  • ✓The interfering chunk is shorter than half the utterance, so the primary speaker is the majority.
  • ✓Gated relative position bias: ri−jr_{i-j} is added to every attention score, and query-dependent gates scale it.
  • ✓BASE 95M on LibriSpeech 960 h; BASE+ and LARGE (317M) on 94k h: Libri-Light 60k + GigaSpeech 10k + VoxPopuli 24k.
  • ✓Led SUPERB; biggest gains on speaker verification, diarization and separation.
← back to Deep Learning roadmap