# WavLM

> WavLM is HuBERT plus denoising. It hears a primary utterance mixed with another speaker or noise, and predicts the primary's k-means units at masked frames. With gated relative position bias and 94k h of data it led SUPERB.

Source: https://www.mlfoundry.in/learn/wavlm  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 7 min read · updated 2026-10-15

**WavLM:** is a self-supervised speech encoder trained by **masked speech denoising and prediction**. Its input is an utterance artificially overlapped with another utterance or with noise. On masked frames it predicts the discrete pseudo-labels of the *original* utterance, HuBERT-style. Introduced by Chen et al. (Microsoft), "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing" (IEEE Journal of Selected Topics in Signal Processing, 2022).

**wav2vec 2.0** and **HuBERT** were built for recognition. They are pre-trained on clean, single-speaker audiobooks, and their objectives reward knowing *what* was said. Many speech tasks need more. Speaker verification and **Speaker diarization** need *who* is speaking, separation needs *which* voice is which, and real recordings contain overlap and noise. WavLM's goal was one encoder for the "full stack": content, speaker, paralinguistics and multi-talker audio.

It keeps HuBERT's recipe (CNN encoder, span masking, cross-entropy on k-means units at masked frames) and changes three things: the input is corrupted by **utterance mixing**, the Transformer uses a **gated relative position bias**, and the data grows to **94k hours** from more varied sources.

## Masked speech denoising

For some utterances in a batch, a random chunk of another utterance, or of noise, is added at a random position with a random energy ratio. The chunk is shorter than half the primary utterance, so the primary speaker is always the majority and "whose content to predict" is well defined. The targets are still the units of the **clean primary**. On a masked frame inside the overlap, the model hears two voices, sees neither directly, and must output the primary speaker's unit. To do that it has to track which voice is which, and that is speaker information.

> Figure: WavLM pre-training. A primary utterance u is mixed with a random chunk of another utterance or noise, shorter than half of u. The mixture passes through the CNN encoder (20 ms frames), span masking and a Transformer with gated relative position bias. On masked frames, the output is trained with cross-entropy against the k-means units of the clean u. Masked frames inside the overlap (orange outline) are where denoising is learned. BASE is trained on LibriSpeech 960 h; BASE+ and LARGE on 94k h.

$$
g^{\text{upd}}_i,\;g^{\text{rst}}_i=\sigma(q_i\!\cdot u),\;\sigma(q_i\!\cdot w),\qquad r_{i-j}=d_{i-j}+g^{\text{upd}}_i\,d_{i-j}+\big(1-g^{\text{upd}}_i\big)\,s\,g^{\text{rst}}_i\,d_{i-j},\qquad a_{ij}\propto\exp\!\Big(\frac{q_i\cdot k_j}{\sqrt{d_k}}+r_{i-j}\Big)
$$

- $d_{i-j}$: learned scalar bias for the (bucketed) relative distance $i-j$, as in T5
- $u,\,w$: learned vectors; the gates depend on the query $q_i$, so each frame can decide how much distance matters
- $s$: learned scalar
- $a_{ij}$: attention weight from frame $i$ to frame $j$ (see **Self-attention**)

wav2vec 2.0 and HuBERT add position through a convolution on the input. A relative bias acts inside every attention score, and the gates let the content of the current frame scale it. For example, a silent or noisy frame can pay less attention to near neighbours. The paper reports that the gated bias improved results over a plain relative bias.

**Models and data (Chen et al., 2022)**

- **BASE:** 12 layers, width 768, about 95M parameters; LibriSpeech 960 h
- **BASE+:** same size as BASE; 94k h
- **LARGE:** 24 layers, width 1024, about 317M parameters; 94k h
- **94k hours:** Libri-Light 60k h (audiobooks) + GigaSpeech 10k h (podcasts, YouTube) + VoxPopuli 24k h (European Parliament), all English
- **Targets:** k-means pseudo-labels in the HuBERT manner, from an earlier model's features
- **SUPERB:** WavLM LARGE ranked first on the SUPERB benchmark at publication; the largest gains were on speaker verification, diarization and separation

> Figure: Utterance mixing on a toy 2 s utterance (100 frames of 20 ms; toy 2 kHz sample rate). Defaults: a second voice covers 30% of it (frames 57–86) at an energy ratio of 5 dB inside the overlap. Over the whole utterance this is 11.27 dB, because the interferer is silent elsewhere. Span masking (p = 0.08, M = 10) hides 53 frames, 11 of them inside the overlap. For 4 of those 11, the toy unit of the mixture differs from the clean unit, and 10 of the 30 overlap frames change overall. Those frames would get wrong targets if units were computed from the mix, which is why WavLM labels the clean primary. At 20 dB only 1 masked frame (2 of 30 overall) changes, and the utterance SNR is 26.27 dB. With noise instead of a voice at 5 dB, it is 5 of 11. Above 50% overlap the figure warns that the main speaker becomes ambiguous.

## Worked example: setting the mix level

**Primary 4 s; interferer chunk of 1.2 s (30% overlap) at a 5 dB energy ratio**

1. Primary energy over the whole utterance $E_p=100$, inside the chosen region $E_p^{\text{reg}}=30$. Energy of the raw interferer chunk $E_s=50$.
2. Gain for a ratio of $r=5$ dB inside the region: $g=\sqrt{E_p^{\text{reg}}/(E_s\cdot10^{r/10})}=\sqrt{30/(50\cdot3.162)}=0.436$.
3. Scaled interferer energy: $g^2E_s=0.190\cdot50=9.49$, so the ratio inside the region is $10\log_{10}(30/9.49)=5$ dB, as intended.
4. Over the whole utterance: $10\log_{10}(100/9.49)=10.23$ dB, which equals $5+10\log_{10}(100/30)$. A short overlap looks mild in utterance SNR but is just as hard locally.
5. Training pair: input = primary + $0.436\cdot$chunk; targets = units of the primary alone, used on masked frames only.

```python
import numpy as np

def mix_utterance(primary, other, rng, max_ratio=0.5, snr_db=(-5.0, 5.0)):
    """WavLM-style utterance mixing: add a random chunk of `other`, shorter than
    max_ratio of the primary, at a random position and energy ratio (dB)."""
    n = len(primary)
    length = rng.integers(1, int(max_ratio * n))              # < half: primary stays the majority
    start = rng.integers(0, n - length + 1)
    src = rng.integers(0, len(other) - length + 1)
    chunk = other[src:src + length]
    region = primary[start:start + length]
    ratio = rng.uniform(*snr_db)
    gain = np.sqrt(np.sum(region**2) / (np.sum(chunk**2) * 10 ** (ratio / 10)))
    mixed = primary.copy()
    mixed[start:start + length] += gain * chunk
    return mixed, (start, length, ratio)

sr = 16_000
t = np.arange(4 * sr) / sr
rng = np.random.default_rng(0)
primary = np.sin(2 * np.pi * 140 * t) * (0.6 + 0.4 * np.sin(2 * np.pi * 3 * t))
other = np.sin(2 * np.pi * 230 * t) * (0.6 + 0.4 * np.cos(2 * np.pi * 2 * t))
mixed, (start, length, ratio) = mix_utterance(primary, other, rng)
interf = mixed - primary
print(f"overlap {length / len(primary):.1%} at {start / sr:.2f} s, ratio {ratio:.2f} dB in the region,",
      f"{10 * np.log10(np.sum(primary**2) / np.sum(interf**2)):.2f} dB over the utterance")
# The model gets `mixed`; its targets are the k-means units of `primary`.
```

WavLM-style utterance mixing in numpy. The 50% cap follows the paper; the −5…5 dB range is illustrative. Prints: overlap 42.5% at 1.46 s, ratio −4.59 dB in the region, −0.90 dB over the utterance (run here).

## Where it sits

| Aspect | wav2vec 2.0 | HuBERT | WavLM |
| --- | --- | --- | --- |
| Input | clean speech | clean speech | speech mixed with speech or noise |
| Targets | jointly learned quantised latents | offline k-means units | offline k-means units of the clean primary |
| Loss | contrastive, 100 distractors | cross-entropy, masked frames | cross-entropy, masked frames |
| Position | conv embedding | conv embedding | conv embedding + gated relative bias |
| Largest data | LV-60k (53.2k h) | Libri-Light 60k h | 94k h, three sources |

In practice WavLM is used like HuBERT: load a checkpoint such as `microsoft/wavlm-base-plus` or `microsoft/wavlm-large` in Hugging Face transformers and read the hidden states. Speaker verification adds an x-vector or ECAPA-style head on top, and diarization pipelines use it to embed short windows.

> **Why denoising teaches speaker identity:** Predicting a masked unit in clean speech can be done from content alone. In a two-voice mixture it cannot: the context contains both voices, and only the primary's units count. The model has to keep each voice's identity apart over time. That is exactly the representation speaker verification and diarization need, and it is why WavLM's gains are largest there, while ASR changes less.

> **Pick the layer per task:** Different layers carry different information. Speaker tasks tend to use lower and middle layers, content tasks higher ones. SUPERB therefore learns a weighted sum over all hidden states for each task. With a single fixed layer, such as the last one, a speaker system built on WavLM can lose much of its advantage.

**Strengths**

- One encoder for content, speaker, emotion and multi-talker tasks; strong on SUPERB
- Robust to overlap and noise by construction
- Same interface as HuBERT: drop-in for existing pipelines

**Limitations**

- Needs an earlier model to produce pseudo-labels, plus a mixing and noise pipeline
- English-only pre-training data; multilingual use needs other checkpoints
- ASR gains over HuBERT are small; the extra cost pays off mainly for speaker and overlap tasks

**Question:** In WavLM's denoising task, what are the targets for a masked frame inside the overlapped region?

- A. The k-means units of the mixture
- B. The k-means units of the clean primary utterance
- C. The units of both speakers, as two outputs
- D. The waveform of the interferer

**Answer:** B. The k-means units of the clean primary utterance — The model sees only the mixture, but is scored against the primary utterance's original pseudo-labels. Ignoring the interfering voice is therefore part of the task, which builds speaker-aware and noise-robust features.

**Question:** Why is the interfering chunk kept shorter than half the primary utterance?

- A. To save compute
- B. So the primary speaker is unambiguously the majority speaker, and 'predict the main speaker' stays well defined
- C. Because masking covers the other half
- D. Because the CNN cannot process longer overlaps

**Answer:** B. So the primary speaker is unambiguously the majority speaker, and 'predict the main speaker' stays well defined — If the interferer covered most of the audio, nothing in the input would say which voice is the 'primary', and the target would be arbitrary. Keeping the overlap under 50% makes the speaker who talks most of the time the one to follow.

**Key points**

- WavLM = HuBERT-style masked unit prediction + denoising: the input is mixed with another utterance or noise, the targets come from the clean primary.
- The interfering chunk is shorter than half the utterance, so the primary speaker is the majority.
- Gated relative position bias: $r_{i-j}$ is added to every attention score, and query-dependent gates scale it.
- BASE 95M on LibriSpeech 960 h; BASE+ and LARGE (317M) on 94k h: Libri-Light 60k + GigaSpeech 10k + VoxPopuli 24k.
- Led SUPERB; biggest gains on speaker verification, diarization and separation.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
