Definition
wav2vec 2.0 is a self-supervised speech encoder. It reads the raw waveform, masks spans of its latent frames, and learns by a contrastive task: for each masked step, identify the true quantised latent among distractors. It was introduced by Baevski, Zhou, Mohamed and Auli, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" (Facebook AI, NeurIPS 2020). After pre-training, the quantiser is dropped and the encoder is fine-tuned with CTC for recognition.
The input is a 16 kHz waveform. Pre-training produces one context vector per 20 ms frame, and fine-tuning maps those vectors to characters. Speech-to-text covers the recognition pipeline. This article covers the pre-training.
Labelled speech is expensive: a transcribed hour costs real annotator time, and most languages have little of it. Unlabelled audio is plentiful. Earlier self-supervised models (CPC, the first wav2vec, vq-wav2vec) predicted future frames, or learned discrete units in a separate first stage and then trained BERT on them. wav2vec 2.0 learns the units and the contextual Transformer jointly, and makes the Transformer use context on both sides by masking, as in BERT.
Architecture
A feature encoder of 7 temporal convolution blocks (512 channels, GELU, normalisation) turns the waveform into latents . Its total stride is 320 samples (20 ms) and its receptive field is 400 samples (25 ms): the window of a classic log-mel front end, with a 20 ms hop instead of 10 ms. Spans of are replaced by one shared, learned mask vector and fed to a Transformer , which outputs context vectors . A convolution with kernel 128 and 16 groups adds relative positional information instead of absolute position embeddings.
Separately, the unmasked latents go to a product quantiser. It has codebooks of entries each. For every frame it picks one entry per codebook, concatenates the two and applies a linear map to get . That gives possible codes from only 640 stored vectors. The choice is made differentiable with a straight-through Gumbel softmax, whose temperature is annealed during training.
Masking and the loss
About frames are chosen at random as span starts, with . Each start masks itself and the following frames, in total. Spans may overlap, so the masked fraction is less than . A frame stays visible only if none of the positions that could cover it was chosen as a start:
- fraction of frames chosen as span starts (0.065)
- span length in frames (10, i.e. 200 ms)
The paper reports about 49% of frames masked, with a mean span length of 14.7 frames, because overlapping spans merge. Sampling reproduces this: 49.0% and 14.66 frames.
- cosine similarity; is the temperature
- plus distractors sampled uniformly from other masked steps of the same utterance
- average softmax probability of entry in codebook over a batch. is minus the entropy and is lowest when all entries are used equally
- weight of the diversity loss, 0.1
is the InfoNCE loss over 101 candidates. Distractors come from the same utterance, so speaker and channel give no clue: the model has to tell apart what was said at different moments. Without the quantiser could collapse onto a few entries, which makes the contrastive task trivial. For the full 640 entries, the minimum of is .
Configurations and results (Baevski et al., 2020)
- Feature encoder
- 7 conv blocks, 512 channels; kernels (10, 3, 3, 3, 3, 2, 2), strides (5, 2, 2, 2, 2, 2, 2)
- BASE
- 12 Transformer blocks, width 768, FFN 3072; about 95M parameters
- LARGE
- 24 blocks, width 1024, FFN 4096; about 317M parameters
- Pre-training data
- LibriSpeech 960 h (LS-960) without transcripts, or LibriVox LV-60k (about 53.2k h after preprocessing)
- Fine-tuning
- linear layer onto characters, CTC loss; Libri-light splits of 10 min, 1 h, 10 h, plus LibriSpeech 100 h and 960 h
- LARGE, LV-60k, test-clean / test-other WER
- 10 min: 4.8 / 8.2 · 1 h: 2.9 / 5.8 · 100 h: 2.0 / 4.0 · 960 h: 1.8 / 3.3 (with a Transformer LM)
Worked example: from samples to frames
| Block | Kernel | Stride | Total stride (samples) | Receptive field (samples) | Output length |
|---|---|---|---|---|---|
| 1 | 10 | 5 | 5 | 10 | 3199 |
| 2 | 3 | 2 | 10 | 20 | 1599 |
| 3 | 3 | 2 | 20 | 40 | 799 |
| 4 | 3 | 2 | 40 | 80 | 399 |
| 5 | 3 | 2 | 80 | 160 | 199 |
| 6 | 2 | 2 | 160 | 240 | 99 |
| 7 | 2 | 2 | 320 = 20 ms | 400 = 25 ms | 49 |
Now one masked step. Its context vector has cosine 0.8 with and 0.5, 0.2, −0.1 with three distractors. At the logits are 8, 5, 2, −1, and the softmax gives the target 0.950, so . At the target gets only 0.371 and : a low temperature sharpens the contrast. If one distractor happens to carry the same code as the target (cosine 0.8 too), the best possible loss is . Here it is 0.719. Neighbouring frames of one phone often share a code, so the loss rarely reaches zero.
1import numpy as np23def compute_mask(T, rng, p=0.065, M=10):4 """wav2vec 2.0 span masking: ~p*T distinct starts, each masks M frames (spans may overlap)."""5 n_starts = int(p * T + rng.random())6 starts = rng.choice(T - M + 1, n_starts, replace=False)7 mask = np.zeros(T, dtype=bool)8 for s in starts:9 mask[s:s + M] = True10 return mask1112def contrastive_loss(c, q, mask, rng, K=100, kappa=0.1):13 """c: (T, d) context vectors, q: (T, d) quantised targets. Mean L_m over masked steps."""14 c = c / np.linalg.norm(c, axis=1, keepdims=True)15 q = q / np.linalg.norm(q, axis=1, keepdims=True)16 idx = np.flatnonzero(mask)17 losses = []18 for t in idx:19 others = idx[idx != t]20 neg = rng.choice(others, min(K, len(others)), replace=False) # distractors: other masked steps21 logits = c[t] @ q[np.r_[t, neg]].T / kappa # target first22 losses.append(np.logaddexp.reduce(logits) - logits[0]) # -log softmax of the target23 return np.mean(losses)2425T, d = 500, 256 # 10 s of audio at 50 frames/s26rng = np.random.default_rng(1)27mask = compute_mask(T, rng)28q = rng.normal(size=(T, d))29c_good = q + 0.5 * rng.normal(size=(T, d)) # context that has learned the targets30c_rand = rng.normal(size=(T, d)) # untrained context31print(f"masked {mask.mean():.1%}")32print(f"L_m trained {contrastive_loss(c_good, q, mask, rng):.3f}, random {contrastive_loss(c_rand, q, mask, rng):.3f}, ln(101) = {np.log(101):.3f}")Use when
- +Learns from raw audio without transcripts; strong WER with very few labels
- +One pre-trained encoder fine-tunes for ASR, keyword spotting, emotion or language ID
- +The 50 Hz frame rate pairs naturally with CTC on characters
Avoid when
- −Pre-training is expensive (LARGE models train for days on many GPUs) and sensitive to hyper-parameters
- −Contrastive targets can collapse without the diversity loss; distractors with the same code act as false negatives
- −Monolingual LibriSpeech/LibriVox checkpoints transfer poorly to noisy or very different domains; HuBERT and WavLM improved on this
Check yourself
With p = 0.065 and M = 10, why are about 49% of frames masked rather than 65%?
Check yourself
Why are the K = 100 distractors drawn from the same utterance?
Remember
- ✓Raw 16 kHz audio → 7-layer CNN (stride 320 = 20 ms, receptive field 25 ms) → latents z → masked → Transformer → context c.
- ✓Span masking with , hides about of frames; mean span 14.7 frames.
- ✓Targets: product quantiser, 2 codebooks × 320 entries, Gumbel softmax; diversity loss (α = 0.1) keeps the codebooks used.
- ✓Loss: InfoNCE over + 100 distractors from other masked steps of the same utterance, cosine / with .
- ✓BASE 95M, LARGE 317M. LARGE on LV-60k with CTC fine-tuning: 4.8/8.2 WER from 10 min of labels, 1.8/3.3 from 960 h.