# wav2vec 2.0

> wav2vec 2.0 masks spans of 20 ms CNN latents and trains a Transformer to pick each hidden step's quantised latent from 100 distractors. Fine-tuned with CTC on 10 minutes of labels it reached 4.8/8.2 LibriSpeech WER.

Source: https://www.mlfoundry.in/learn/wav2vec-2-0  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**wav2vec 2.0:** is a self-supervised speech encoder. It reads the raw waveform, masks spans of its latent frames, and learns by a **contrastive** task: for each masked step, identify the true **quantised** latent among distractors. It was introduced by Baevski, Zhou, Mohamed and Auli, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" (Facebook AI, NeurIPS 2020). After pre-training, the quantiser is dropped and the encoder is fine-tuned with CTC for recognition.

The input is a 16 kHz waveform. Pre-training produces one context vector per 20 ms frame, and fine-tuning maps those vectors to characters. **Speech-to-text** covers the recognition pipeline. This article covers the pre-training.

Labelled speech is expensive: a transcribed hour costs real annotator time, and most languages have little of it. Unlabelled audio is plentiful. Earlier self-supervised models (CPC, the first wav2vec, vq-wav2vec) predicted *future* frames, or learned discrete units in a separate first stage and then trained BERT on them. wav2vec 2.0 learns the units and the contextual Transformer **jointly**, and makes the Transformer use context on both sides by masking, as in **BERT**.

## Architecture

A **feature encoder** $f$ of 7 temporal convolution blocks (512 channels, GELU, normalisation) turns the waveform into latents $z_1,\dots,z_T$. Its total stride is 320 samples (20 ms) and its receptive field is 400 samples (25 ms): the window of a classic log-mel front end, with a 20 ms hop instead of 10 ms. Spans of $z$ are replaced by one shared, learned mask vector and fed to a **Transformer** $g$, which outputs context vectors $c_t$. A convolution with kernel 128 and 16 groups adds relative positional information instead of absolute position embeddings.

Separately, the **unmasked** latents go to a **product quantiser**. It has $G=2$ codebooks of $V=320$ entries each. For every frame it picks one entry per codebook, concatenates the two and applies a linear map to get $q_t$. That gives $320^2=102{,}400$ possible codes from only 640 stored vectors. The choice is made differentiable with a straight-through **Gumbel softmax**, whose temperature is annealed during training.

> Figure: wav2vec 2.0 pre-training. The 7-layer CNN turns 16 kHz audio into 512-d latents at 50 per second (stride 320 = 20 ms, receptive field 400 samples = 25 ms). The quantiser sees the unmasked latents and produces targets q (2 codebooks × 320 entries). The Transformer sees the masked sequence and produces c. The loss is computed at masked steps only: c_t must pick q_t out of 100 distractors from other masked steps, and a diversity term keeps all codebook entries in use. For fine-tuning, the quantiser is removed and a linear CTC layer is added.

## Masking and the loss

About $pT$ frames are chosen at random as span **starts**, with $p=0.065$. Each start masks itself and the following frames, $M=10$ in total. Spans may overlap, so the masked fraction is less than $pM=65\%$. A frame stays visible only if none of the $M$ positions that could cover it was chosen as a start:

$$
P(\text{masked})\approx1-(1-p)^{M}=1-0.935^{10}=0.489
$$

- $p$: fraction of frames chosen as span starts (0.065)
- $M$: span length in frames (10, i.e. 200 ms)

The paper reports about 49% of frames masked, with a mean span length of 14.7 frames, because overlapping spans merge. Sampling reproduces this: 49.0% and 14.66 frames.

$$
\mathcal L_m=-\log\frac{\exp\!\big(\mathrm{sim}(c_t,q_t)/\kappa\big)}{\sum_{\tilde q\in Q_t}\exp\!\big(\mathrm{sim}(c_t,\tilde q)/\kappa\big)},\qquad \mathcal L_d=\frac{1}{GV}\sum_{g=1}^{G}\sum_{v=1}^{V}\bar p_{g,v}\log\bar p_{g,v},\qquad \mathcal L=\mathcal L_m+\alpha\,\mathcal L_d
$$

- $\mathrm{sim}$: cosine similarity; $\kappa=0.1$ is the temperature
- $Q_t$: $q_t$ plus $K=100$ distractors sampled uniformly from **other masked steps of the same utterance**
- $\bar p_{g,v}$: average softmax probability of entry $v$ in codebook $g$ over a batch. $\mathcal L_d$ is minus the entropy and is lowest when all entries are used equally
- $\alpha$: weight of the diversity loss, 0.1

$\mathcal L_m$ is the **InfoNCE loss** over 101 candidates. Distractors come from the same utterance, so speaker and channel give no clue: the model has to tell apart what was said at different moments. Without $\mathcal L_d$ the quantiser could collapse onto a few entries, which makes the contrastive task trivial. For the full 640 entries, the minimum of $\mathcal L_d$ is $-\ln 320/320=-0.018$.

**Configurations and results (Baevski et al., 2020)**

- **Feature encoder:** 7 conv blocks, 512 channels; kernels (10, 3, 3, 3, 3, 2, 2), strides (5, 2, 2, 2, 2, 2, 2)
- **BASE:** 12 Transformer blocks, width 768, FFN 3072; about 95M parameters
- **LARGE:** 24 blocks, width 1024, FFN 4096; about 317M parameters
- **Pre-training data:** LibriSpeech 960 h (LS-960) without transcripts, or LibriVox LV-60k (about 53.2k h after preprocessing)
- **Fine-tuning:** linear layer onto characters, **CTC** loss; Libri-light splits of 10 min, 1 h, 10 h, plus LibriSpeech 100 h and 960 h
- **LARGE, LV-60k, test-clean / test-other WER:** 10 min: 4.8 / 8.2 · 1 h: 2.9 / 5.8 · 100 h: 2.0 / 4.0 · 960 h: 1.8 / 3.3 (with a Transformer LM)

> Figure: Span masking and contrastive scoring on a toy 3 s utterance (150 frames). At the paper's settings (p = 0.065, M = 10) the formula gives 48.9% and sampling 300 ten-second utterances gives 49.0%, with mean span 14.66 frames (293 ms). The dashed line p·M = 65% ignores overlaps. This utterance has 81 masked frames in 5 spans, so K is capped at 80. Frame 22 (depth 3 in its span) has cosine 0.680 with its target and ranks first. Two distractors share its code and tie with it, and L_t = 1.613. Mean L_m is 2.707, against a chance level of ln 81 = 4.394. It is 1.467 at span edges and 3.626 five or more frames deep, because deep frames have less nearby context. Set M = 20: 73.9% masked, mean span 43 frames, and top-1 accuracy drops from 76.5% to 60.7%. Set κ = 1: the softmax flattens and mean L_m rises to 3.861.

## Worked example: from samples to frames

1 s of 16 kHz audio (16,000 samples) through the feature encoder. Receptive field grows by (kernel − 1) × the incoming stride. Computed in Python.

| Block | Kernel | Stride | Total stride (samples) | Receptive field (samples) | Output length |
| --- | --- | --- | --- | --- | --- |
| 1 | 10 | 5 | 5 | 10 | 3199 |
| 2 | 3 | 2 | 10 | 20 | 1599 |
| 3 | 3 | 2 | 20 | 40 | 799 |
| 4 | 3 | 2 | 40 | 80 | 399 |
| 5 | 3 | 2 | 80 | 160 | 199 |
| 6 | 2 | 2 | 160 | 240 | 99 |
| 7 | 2 | 2 | 320 = 20 ms | 400 = 25 ms | 49 |

Now one masked step. Its context vector has cosine 0.8 with $q_t$ and 0.5, 0.2, −0.1 with three distractors. At $\kappa=0.1$ the logits are 8, 5, 2, −1, and the softmax gives the target 0.950, so $\mathcal L_m=0.051$. At $\kappa=1$ the target gets only 0.371 and $\mathcal L_m=0.992$: a low temperature sharpens the contrast. If one distractor happens to carry the same code as the target (cosine 0.8 too), the best possible loss is $\ln 2$. Here it is 0.719. Neighbouring frames of one phone often share a code, so the loss rarely reaches zero.

```python
import numpy as np

def compute_mask(T, rng, p=0.065, M=10):
    """wav2vec 2.0 span masking: ~p*T distinct starts, each masks M frames (spans may overlap)."""
    n_starts = int(p * T + rng.random())
    starts = rng.choice(T - M + 1, n_starts, replace=False)
    mask = np.zeros(T, dtype=bool)
    for s in starts:
        mask[s:s + M] = True
    return mask

def contrastive_loss(c, q, mask, rng, K=100, kappa=0.1):
    """c: (T, d) context vectors, q: (T, d) quantised targets. Mean L_m over masked steps."""
    c = c / np.linalg.norm(c, axis=1, keepdims=True)
    q = q / np.linalg.norm(q, axis=1, keepdims=True)
    idx = np.flatnonzero(mask)
    losses = []
    for t in idx:
        others = idx[idx != t]
        neg = rng.choice(others, min(K, len(others)), replace=False)   # distractors: other masked steps
        logits = c[t] @ q[np.r_[t, neg]].T / kappa                     # target first
        losses.append(np.logaddexp.reduce(logits) - logits[0])         # -log softmax of the target
    return np.mean(losses)

T, d = 500, 256                                          # 10 s of audio at 50 frames/s
rng = np.random.default_rng(1)
mask = compute_mask(T, rng)
q = rng.normal(size=(T, d))
c_good = q + 0.5 * rng.normal(size=(T, d))               # context that has learned the targets
c_rand = rng.normal(size=(T, d))                         # untrained context
print(f"masked {mask.mean():.1%}")
print(f"L_m trained {contrastive_loss(c_good, q, mask, rng):.3f}, random {contrastive_loss(c_rand, q, mask, rng):.3f}, ln(101) = {np.log(101):.3f}")
```

wav2vec 2.0 span masking and the masked contrastive loss in numpy. Prints masked 46.4%, then L_m 0.016 for a context that matches its targets and 4.856 for a random one, near ln 101 = 4.615 (run here).

> **Fine-tuning details matter:** Fine-tuning keeps the CNN feature encoder frozen and trains only the Transformer and the new output layer. The Transformer itself is also frozen for the first updates, so the random output layer can settle first. Masking stays on during fine-tuning, as a SpecAugment-like regulariser in time and channels. With 10 minutes of labels, skipping these steps quickly overfits.

> **Why quantise the targets but not the input:** The Transformer reads **continuous** latents, which keep all the detail. The targets are **discrete**, which removes detail the model should not have to predict, such as exact phase and noise. Contrasting against quantised targets worked better in the paper's ablations than continuous targets, or than quantising the inputs as well.

**Strengths**

- Learns from raw audio without transcripts; strong WER with very few labels
- One pre-trained encoder fine-tunes for ASR, keyword spotting, emotion or language ID
- The 50 Hz frame rate pairs naturally with CTC on characters

**Limitations**

- Pre-training is expensive (LARGE models train for days on many GPUs) and sensitive to hyper-parameters
- Contrastive targets can collapse without the diversity loss; distractors with the same code act as false negatives
- Monolingual LibriSpeech/LibriVox checkpoints transfer poorly to noisy or very different domains; **HuBERT** and **WavLM** improved on this

**Question:** With p = 0.065 and M = 10, why are about 49% of frames masked rather than 65%?

- A. Spans that would run past the end of the utterance are dropped
- B. Spans overlap: a frame is masked if any of the 10 starts that could cover it is chosen, so P = 1 − 0.935¹⁰ ≈ 0.489
- C. Only half of the utterances in a batch are masked
- D. The quantiser removes 16% of frames

**Answer:** B. Spans overlap: a frame is masked if any of the 10 starts that could cover it is chosen, so P = 1 − 0.935¹⁰ ≈ 0.489 — $pM$ counts covered frames as if spans never overlapped. Starts are random, so spans do overlap and merge. A frame is visible only if all $M$ positions before it (itself included) are non-starts, with probability $(1-p)^M$. Merging is also why the mean span is 14.7 frames, not 10.

**Question:** Why are the K = 100 distractors drawn from the same utterance?

- A. It is cheaper to store
- B. So that speaker and channel cannot separate target from distractor, forcing the model to encode content
- C. Because other utterances have different lengths
- D. So that the diversity loss is zero

**Answer:** B. So that speaker and channel cannot separate target from distractor, forcing the model to encode content — Distractors from other recordings could be rejected just by voice or microphone characteristics. Within one utterance, these are constant, so the model must tell what was said at each moment, which is what transfers to recognition.

**Key points**

- Raw 16 kHz audio → 7-layer CNN (stride 320 = 20 ms, receptive field 25 ms) → latents z → masked → Transformer → context c.
- Span masking with $p=0.065$, $M=10$ hides about $1-(1-p)^M\approx49\%$ of frames; mean span 14.7 frames.
- Targets: product quantiser, 2 codebooks × 320 entries, Gumbel softmax; diversity loss (α = 0.1) keeps the codebooks used.
- Loss: InfoNCE over $q_t$ + 100 distractors from other masked steps of the same utterance, cosine / $\kappa$ with $\kappa=0.1$.
- BASE 95M, LARGE 317M. LARGE on LV-60k with CTC fine-tuning: 4.8/8.2 WER from 10 min of labels, 1.8/3.3 from 960 h.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
