Definition
WaveNet is an autoregressive generative model of raw audio waveforms. It factorises and models each conditional with a deep stack of dilated causal convolutions ending in a softmax over 256 quantised amplitude values. Introduced by van den Oord, Dieleman, Zen, Simonyan, Vinyals, Graves, Kalchbrenner, Senior and Kavukcuoglu (DeepMind), "WaveNet: A Generative Model for Raw Audio" (2016).
The task: given past samples, and optionally a speaker or linguistic features, output the next sample. Audio is extremely dense. One second at 16 kHz is 16,000 values, and speech has structure at many time scales at once: pitch periods of a few milliseconds, phonemes of tens of milliseconds and words lasting hundreds of milliseconds.
Before WaveNet, speech synthesis either concatenated recorded units or ran a hand-designed signal-processing vocoder on predicted acoustic parameters (see Text-to-speech). Both were limited by the vocoder or by the unit database. WaveNet removed the vocoder: the network itself is the waveform model. It applies the idea of PixelCNN (see Autoregressive generative models) to one-dimensional sound.
Dilated causal convolutions
A causal convolution only reads , so the model can be trained in parallel with teacher forcing and still be sampled left to right. With kernel size 2 and no dilation, each layer adds only one sample of context, so covering 100 ms (1,600 samples at 16 kHz) would take about 1,600 layers. A dilated convolution (see Dilated convolution) skips inputs between taps. Doubling at every layer makes the receptive field grow exponentially with depth.
- receptive field in samples: how far back the output at can see
- kernel size (2 in WaveNet) and dilation of layer
- number of repeated 10-layer blocks. One block gives 1,024 samples (64 ms at 16 kHz); the paper's example of three blocks gives 3,070 samples (192 ms)
Gates, residuals and skips
- dilated convolution filters for the filter and the gate branch of one layer; is convolution
- element-wise product and logistic sigmoid. This gated unit, taken from PixelCNN, worked better than ReLU for audio in the paper
- conditioning. Global: one vector per utterance, such as a speaker embedding. Local: a time series such as linguistic features or a mel spectrogram, first upsampled to the audio rate (with a transposed convolution or by repetition), then added through a 1×1 convolution
Each layer's output passes through a 1×1 convolution. The result is added to the layer input (a residual connection) and is also sent to a separate skip path. The output head sums the skips of all layers and applies ReLU, 1×1, ReLU, 1×1 and a softmax. Residual and skip paths let a stack of dozens of layers train, and let the head see features from every time scale.
μ-law: 256 classes instead of 65,536
WaveNet treats each sample as a classification target. A categorical softmax makes no assumption about the shape of the distribution, so it can be multi-modal. Raw 16-bit audio would need 65,536 classes per step, however. The paper first compresses amplitudes with μ-law companding and then quantises to 8 bits:
- compressed amplitude in , then cut into 256 uniform bins. Back in , the bins next to zero are 0.00017 wide and those at full scale 0.043, about 245 times wider
- compression strength; 255 is the value used by G.711 telephony and by WaveNet
Uniform 8-bit steps are wide everywhere. Quiet passages, which are common in speech, would then use only a handful of levels. μ-law spends its bins where the samples are, so the relative error stays roughly constant across loudness. The authors report that the 8-bit μ-law reconstruction sounded very similar to the original.
| x | F(x) | μ-law class | decoded | |error| μ-law | |error| linear |
|---|---|---|---|---|---|
| 0.5 | 0.8757 | 240 | 0.50903 | 0.00903 | 0.00391 |
| −0.2 | −0.7126 | 36 | −0.20260 | 0.00260 | 0.00078 |
| 0.05 | 0.4727 | 188 | 0.05000 | 0.00000 | 0.00078 |
| 0.01 | 0.2285 | 157 | 0.01015 | 0.00015 | 0.00172 |
| 0.001 | 0.0410 | 133 | 0.00106 | 0.00006 | 0.00291 |
Loud samples lose a little precision (the μ-law bin containing 0.5 is 0.022 wide), while quiet ones gain a lot (the bin at 0.001 is 0.00022 wide). At 0.001, linear 8-bit cannot tell the sample from zero: it lands in the first bin above zero, whose centre is 0.0039.
From the paper (van den Oord et al., 2016)
- Dilations
- 1, 2, 4, …, 512, repeated (the paper's example: three times)
- Output
- 256-way softmax over 8-bit μ-law values (μ = 255)
- Multi-speaker
- VCTK, 44 hours from 109 speakers, with a one-hot speaker ID as global conditioning
- TTS data
- single-speaker North American English (24.6 h) and Mandarin Chinese, conditioned on linguistic features plus log F0
- US English MOS
- WaveNet 4.21 vs 3.86 (HMM-driven unit selection) and 3.67 (LSTM parametric); natural 16-bit speech 4.55
Sampling is slow
Training is parallel, because every target is known. Generation is not. Sample must be drawn and fed back before sample can be computed, so one second of 16 kHz audio needs 16,000 sequential passes through the whole stack. Caching activations of earlier steps removes recomputation, but the chain of dependencies remains.
Parallel WaveNet (van den Oord et al., 2018) trained a non-autoregressive inverse-autoregressive-flow student to match a trained WaveNet teacher (probability density distillation, a form of Knowledge distillation). The student generates all samples at once and ran more than 20 times faster than real time at 24 kHz. WaveRNN attacked the same cost with a small recurrent network, and GAN vocoders such as HiFi-GAN removed autoregression altogether. WaveNet also became the vocoder of Tacotron 2, conditioned on mel spectrograms.
1import numpy as np23def mu_law_encode(x, mu=255): # x in (-1, 1) -> integer class 0..2554 y = np.sign(x) * np.log1p(mu * np.abs(x)) / np.log1p(mu)5 return np.clip(((y + 1) / 2 * (mu + 1)).astype(int), 0, mu)67def mu_law_decode(q, mu=255): # class -> bin centre, expanded back8 y = 2 * (q + 0.5) / (mu + 1) - 19 return np.sign(y) * ((1 + mu) ** np.abs(y) - 1) / mu1011sr = 1600012x = 0.3 * np.sin(2 * np.pi * 440 * np.arange(sr) / sr)13q = mu_law_encode(x) # the training targets for the softmax14xh = mu_law_decode(q)15print(q.min(), q.max(), 10 * np.log10(np.sum(x**2) / np.sum((x - xh)**2))) # 27 228 38.11617dilations = [2**i for i in range(10)] * 3 # 30 layers, kernel size 218R = 1 + sum(dilations)19print(R, 1000 * R / sr, "ms") # 3070 191.875 msCheck yourself
A WaveNet uses kernel size 2 and dilations 1, 2, …, 512 repeated twice (20 layers). What is its receptive field at 16 kHz?
Check yourself
Why does WaveNet quantise with μ-law rather than uniformly to 8 bits?
Remember
- ✓Autoregressive over raw samples: ; training is parallel, sampling is sequential.
- ✓Dilated causal convolutions, repeated: receptive field samples (3,070 for , 192 ms at 16 kHz).
- ✓Gated unit , with residual and skip connections; global (speaker) and local (features, mel) conditioning.
- ✓8-bit μ-law () → a 256-way softmax that can be multi-modal.
- ✓Sampling cost motivated Parallel WaveNet, WaveRNN and GAN vocoders such as HiFi-GAN.