wav2vec 2.0

wav2vec 2.0 masks spans of 20 ms CNN latents and trains a Transformer to pick each hidden step's quantised latent from 100 distractors. Fine-tuned with CTC on 10 minutes of labels it reached 4.8/8.2 LibriSpeech WER.

8 min read779 wordscore0 viewsupdated 2026-10-15

Definition

wav2vec 2.0 is a self-supervised speech encoder. It reads the raw waveform, masks spans of its latent frames, and learns by a contrastive task: for each masked step, identify the true quantised latent among distractors. It was introduced by Baevski, Zhou, Mohamed and Auli, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" (Facebook AI, NeurIPS 2020). After pre-training, the quantiser is dropped and the encoder is fine-tuned with CTC for recognition.

The input is a 16 kHz waveform. Pre-training produces one context vector per 20 ms frame, and fine-tuning maps those vectors to characters. Speech-to-text covers the recognition pipeline. This article covers the pre-training.

Labelled speech is expensive: a transcribed hour costs real annotator time, and most languages have little of it. Unlabelled audio is plentiful. Earlier self-supervised models (CPC, the first wav2vec, vq-wav2vec) predicted future frames, or learned discrete units in a separate first stage and then trained BERT on them. wav2vec 2.0 learns the units and the contextual Transformer jointly, and makes the Transformer use context on both sides by masking, as in BERT.

Architecture

A feature encoder ff of 7 temporal convolution blocks (512 channels, GELU, normalisation) turns the waveform into latents z1,…,zTz_1,\dots,z_T. Its total stride is 320 samples (20 ms) and its receptive field is 400 samples (25 ms): the window of a classic log-mel front end, with a 20 ms hop instead of 10 ms. Spans of zz are replaced by one shared, learned mask vector and fed to a Transformer gg, which outputs context vectors ctc_t. A convolution with kernel 128 and 16 groups adds relative positional information instead of absolute position embeddings.

Separately, the unmasked latents go to a product quantiser. It has G=2G=2 codebooks of V=320V=320 entries each. For every frame it picks one entry per codebook, concatenates the two and applies a linear map to get qtq_t. That gives 3202=102,400320^2=102{,}400 possible codes from only 640 stored vectors. The choice is made differentiable with a straight-through Gumbel softmax, whose temperature is annealed during training.

pre-training on unlabelled audio · 16 kHz input · numbers from Baevski et al. (2020)raw waveform xfeature encoder f7 conv blocks, 512 chkernels 10,3,3,3,3,2,2strides 5,2,2,2,2,2,2latents Z · 512-d · 50 per secondz1z2z3z4z5z6z7z8z9z10z11z12stride 320 = 20 ms · receptive field 400 = 25 msquantisation module (unmasked Z)G = 2 codebooks × V = 320 entriesGumbel softmax picks one entry per groupconcat → linear → q_t320² = 102,400 possible codesmaskz1z2z3MMMMz8MMMMp = 0.065, M = 10 (spans drawn as 4)Transformer context network gconv relative position embeddingBASE 12 × 768 · LARGE 24 × 1024c1c2c3c4c5c6c7c8c9c10c11c12q1q2q3q4q5q6q7q8q9q10q11q12loss on masked steps only: L = L_m + α·L_d, α = 0.1L_m: for masked t, pick q_t among q_t + K = 100 distractors drawn from other masked stepsof the same utterance, by cosine similarity / κ (κ = 0.1)L_d: entropy of average codebook use, pushes all 2 × 320 entries into usefine-tuning: drop the quantiser, add a linear layer over characters, train with CTC
fig.wav2vec 2.0 pre-training. The 7-layer CNN turns 16 kHz audio into 512-d latents at 50 per second (stride 320 = 20 ms, receptive field 400 samples = 25 ms). The quantiser sees the unmasked latents and produces targets q (2 codebooks × 320 entries). The Transformer sees the masked sequence and produces c. The loss is computed at masked steps only: c_t must pick q_t out of 100 distractors from other masked steps, and a diversity term keeps all codebook entries in use. For fine-tuning, the quantiser is removed and a linear CTC layer is added.

Masking and the loss

About pTpT frames are chosen at random as span starts, with p=0.065p=0.065. Each start masks itself and the following frames, M=10M=10 in total. Spans may overlap, so the masked fraction is less than pM=65%pM=65\%. A frame stays visible only if none of the MM positions that could cover it was chosen as a start:

P(masked)≈1−(1−p)M=1−0.93510=0.489P(\text{masked})\approx1-(1-p)^{M}=1-0.935^{10}=0.489
pp
fraction of frames chosen as span starts (0.065)
MM
span length in frames (10, i.e. 200 ms)

The paper reports about 49% of frames masked, with a mean span length of 14.7 frames, because overlapping spans merge. Sampling reproduces this: 49.0% and 14.66 frames.

Lm=−log⁡exp⁡ ⁣(sim(ct,qt)/κ)∑q~∈Qtexp⁡ ⁣(sim(ct,q~)/κ),Ld=1GV∑g=1G∑v=1Vpˉg,vlog⁡pˉg,v,L=Lm+α Ld\mathcal L_m=-\log\frac{\exp\!\big(\mathrm{sim}(c_t,q_t)/\kappa\big)}{\sum_{\tilde q\in Q_t}\exp\!\big(\mathrm{sim}(c_t,\tilde q)/\kappa\big)},\qquad \mathcal L_d=\frac{1}{GV}\sum_{g=1}^{G}\sum_{v=1}^{V}\bar p_{g,v}\log\bar p_{g,v},\qquad \mathcal L=\mathcal L_m+\alpha\,\mathcal L_d
sim\mathrm{sim}
cosine similarity; κ=0.1\kappa=0.1 is the temperature
QtQ_t
qtq_t plus K=100K=100 distractors sampled uniformly from other masked steps of the same utterance
pˉg,v\bar p_{g,v}
average softmax probability of entry vv in codebook gg over a batch. Ld\mathcal L_d is minus the entropy and is lowest when all entries are used equally
α\alpha
weight of the diversity loss, 0.1

Lm\mathcal L_m is the InfoNCE loss over 101 candidates. Distractors come from the same utterance, so speaker and channel give no clue: the model has to tell apart what was said at different moments. Without Ld\mathcal L_d the quantiser could collapse onto a few entries, which makes the contrastive task trivial. For the full 640 entries, the minimum of Ld\mathcal L_d is −ln⁡320/320=−0.018-\ln 320/320=-0.018.

Configurations and results (Baevski et al., 2020)

Feature encoder
7 conv blocks, 512 channels; kernels (10, 3, 3, 3, 3, 2, 2), strides (5, 2, 2, 2, 2, 2, 2)
BASE
12 Transformer blocks, width 768, FFN 3072; about 95M parameters
LARGE
24 blocks, width 1024, FFN 4096; about 317M parameters
Pre-training data
LibriSpeech 960 h (LS-960) without transcripts, or LibriVox LV-60k (about 53.2k h after preprocessing)
Fine-tuning
linear layer onto characters, CTC loss; Libri-light splits of 10 min, 1 h, 10 h, plus LibriSpeech 100 h and 960 h
LARGE, LV-60k, test-clean / test-other WER
10 min: 4.8 / 8.2 · 1 h: 2.9 / 5.8 · 100 h: 2.0 / 4.0 · 960 h: 1.8 / 3.3 (with a Transformer LM)
span mask · T = 150 frames (3 s at 50 frames/s) · p = 0.065, M = 100 s1 s2 s3 sexpected 1 − (1 − p)^M = 48.9% · this utterance 81/150 = 54.0% in 5 spans, mean 16.2 framessampled, 300 utterances of 10 s: 49.0% masked, mean span 14.66 frames = 293 msmasked fraction vs M (p fixed)00.51151015201 − (1 − p)^M- - p·M (no overlap)● sampledframe 22: cos to target + 80 distractors-0.500.51targetsame codebeats targetotherframe 22 · code 26 · depth 3 · cos to target 0.680 · rank 1 · 2 same-code · L_t = 1.613mean L_m over 81 masked frames = 2.707 · chance ln(K′+1) = 4.394 · top-1 76.5%mean L_t at depth 1: 1.467 · depth ≥ 5: 3.626 · avg 7.4 same-code distractors per frameK′ = min(K, masked − 1): distractors come only from other masked frames of the same utterancetoy: 32 codes, 16-d · darker cells lie deeper in a span · c_t = target + noise that grows with depth
utterance #1
fig.Span masking and contrastive scoring on a toy 3 s utterance (150 frames). At the paper's settings (p = 0.065, M = 10) the formula gives 48.9% and sampling 300 ten-second utterances gives 49.0%, with mean span 14.66 frames (293 ms). The dashed line p·M = 65% ignores overlaps. This utterance has 81 masked frames in 5 spans, so K is capped at 80. Frame 22 (depth 3 in its span) has cosine 0.680 with its target and ranks first. Two distractors share its code and tie with it, and L_t = 1.613. Mean L_m is 2.707, against a chance level of ln 81 = 4.394. It is 1.467 at span edges and 3.626 five or more frames deep, because deep frames have less nearby context. Set M = 20: 73.9% masked, mean span 43 frames, and top-1 accuracy drops from 76.5% to 60.7%. Set κ = 1: the softmax flattens and mean L_m rises to 3.861.

Worked example: from samples to frames

BlockKernelStrideTotal stride (samples)Receptive field (samples)Output length
11055103199
23210201599
3322040799
4324080399
53280160199
62216024099
722320 = 20 ms400 = 25 ms49
1 s of 16 kHz audio (16,000 samples) through the feature encoder. Receptive field grows by (kernel − 1) × the incoming stride. Computed in Python.

Now one masked step. Its context vector has cosine 0.8 with qtq_t and 0.5, 0.2, −0.1 with three distractors. At κ=0.1\kappa=0.1 the logits are 8, 5, 2, −1, and the softmax gives the target 0.950, so Lm=0.051\mathcal L_m=0.051. At κ=1\kappa=1 the target gets only 0.371 and Lm=0.992\mathcal L_m=0.992: a low temperature sharpens the contrast. If one distractor happens to carry the same code as the target (cosine 0.8 too), the best possible loss is ln⁡2\ln 2. Here it is 0.719. Neighbouring frames of one phone often share a code, so the loss rarely reaches zero.

python
1import numpy as np
2
3def compute_mask(T, rng, p=0.065, M=10):
4 """wav2vec 2.0 span masking: ~p*T distinct starts, each masks M frames (spans may overlap)."""
5 n_starts = int(p * T + rng.random())
6 starts = rng.choice(T - M + 1, n_starts, replace=False)
7 mask = np.zeros(T, dtype=bool)
8 for s in starts:
9 mask[s:s + M] = True
10 return mask
11
12def contrastive_loss(c, q, mask, rng, K=100, kappa=0.1):
13 """c: (T, d) context vectors, q: (T, d) quantised targets. Mean L_m over masked steps."""
14 c = c / np.linalg.norm(c, axis=1, keepdims=True)
15 q = q / np.linalg.norm(q, axis=1, keepdims=True)
16 idx = np.flatnonzero(mask)
17 losses = []
18 for t in idx:
19 others = idx[idx != t]
20 neg = rng.choice(others, min(K, len(others)), replace=False) # distractors: other masked steps
21 logits = c[t] @ q[np.r_[t, neg]].T / kappa # target first
22 losses.append(np.logaddexp.reduce(logits) - logits[0]) # -log softmax of the target
23 return np.mean(losses)
24
25T, d = 500, 256 # 10 s of audio at 50 frames/s
26rng = np.random.default_rng(1)
27mask = compute_mask(T, rng)
28q = rng.normal(size=(T, d))
29c_good = q + 0.5 * rng.normal(size=(T, d)) # context that has learned the targets
30c_rand = rng.normal(size=(T, d)) # untrained context
31print(f"masked {mask.mean():.1%}")
32print(f"L_m trained {contrastive_loss(c_good, q, mask, rng):.3f}, random {contrastive_loss(c_rand, q, mask, rng):.3f}, ln(101) = {np.log(101):.3f}")
wav2vec 2.0 span masking and the masked contrastive loss in numpy. Prints masked 46.4%, then L_m 0.016 for a context that matches its targets and 4.856 for a random one, near ln 101 = 4.615 (run here).

Use when

  • +Learns from raw audio without transcripts; strong WER with very few labels
  • +One pre-trained encoder fine-tunes for ASR, keyword spotting, emotion or language ID
  • +The 50 Hz frame rate pairs naturally with CTC on characters

Avoid when

  • −Pre-training is expensive (LARGE models train for days on many GPUs) and sensitive to hyper-parameters
  • −Contrastive targets can collapse without the diversity loss; distractors with the same code act as false negatives
  • −Monolingual LibriSpeech/LibriVox checkpoints transfer poorly to noisy or very different domains; HuBERT and WavLM improved on this

Check yourself

With p = 0.065 and M = 10, why are about 49% of frames masked rather than 65%?

Check yourself

Why are the K = 100 distractors drawn from the same utterance?

Remember

  • ✓Raw 16 kHz audio → 7-layer CNN (stride 320 = 20 ms, receptive field 25 ms) → latents z → masked → Transformer → context c.
  • ✓Span masking with p=0.065p=0.065, M=10M=10 hides about 1−(1−p)M≈49%1-(1-p)^M\approx49\% of frames; mean span 14.7 frames.
  • ✓Targets: product quantiser, 2 codebooks × 320 entries, Gumbel softmax; diversity loss (α = 0.1) keeps the codebooks used.
  • ✓Loss: InfoNCE over qtq_t + 100 distractors from other masked steps of the same utterance, cosine / κ\kappa with κ=0.1\kappa=0.1.
  • ✓BASE 95M, LARGE 317M. LARGE on LV-60k with CTC fine-tuning: 4.8/8.2 WER from 10 min of labels, 1.8/3.3 from 960 h.
← back to Deep Learning roadmap