WaveNet

WaveNet predicts raw audio one sample at a time: a 256-way softmax over μ-law classes (μ = 255), fed by stacks of gated, dilated causal convolutions (d = 1…512, repeated) whose receptive field grows exponentially with depth.

8 min read798 wordscore0 viewsupdated 2026-10-15

Definition

WaveNet is an autoregressive generative model of raw audio waveforms. It factorises p(x)=∏tp(xt∣x1,…,xt−1)p(x)=\prod_t p(x_t\mid x_1,\dots,x_{t-1}) and models each conditional with a deep stack of dilated causal convolutions ending in a softmax over 256 quantised amplitude values. Introduced by van den Oord, Dieleman, Zen, Simonyan, Vinyals, Graves, Kalchbrenner, Senior and Kavukcuoglu (DeepMind), "WaveNet: A Generative Model for Raw Audio" (2016).

The task: given past samples, and optionally a speaker or linguistic features, output the next sample. Audio is extremely dense. One second at 16 kHz is 16,000 values, and speech has structure at many time scales at once: pitch periods of a few milliseconds, phonemes of tens of milliseconds and words lasting hundreds of milliseconds.

Before WaveNet, speech synthesis either concatenated recorded units or ran a hand-designed signal-processing vocoder on predicted acoustic parameters (see Text-to-speech). Both were limited by the vocoder or by the unit database. WaveNet removed the vocoder: the network itself is the waveform model. It applies the idea of PixelCNN (see Autoregressive generative models) to one-dimensional sound.

Dilated causal convolutions

A causal convolution only reads x≤tx_{\le t}, so the model can be trained in parallel with teacher forcing and still be sampled left to right. With kernel size 2 and no dilation, each layer adds only one sample of context, so covering 100 ms (1,600 samples at 16 kHz) would take about 1,600 layers. A dilated convolution (see Dilated convolution) skips d−1d-1 inputs between taps. Doubling dd at every layer makes the receptive field grow exponentially with depth.

R  =  1+∑l=1L(k−1) dl,dl∈{1,2,4,…,512} repeated B times  ⇒  R=1+1023 B  (k=2)R \;=\; 1+\sum_{l=1}^{L}(k-1)\,d_l,\qquad d_l\in\{1,2,4,\dots,512\}\ \text{repeated }B\text{ times}\;\Rightarrow\;R=1+1023\,B\ \ (k=2)
RR
receptive field in samples: how far back the output at tt can see
k, dlk,\ d_l
kernel size (2 in WaveNet) and dilation of layer ll
BB
number of repeated 10-layer blocks. One block gives 1,024 samples (64 ms at 16 kHz); the paper's example of three blocks gives 3,070 samples (192 ms)
dilated causal convolutions, kernel size 2each layer sees x[t] and x[t − d]; never the futureinput xd = 1d = 2d = 4d = 8 outoutput at t predicts x[t+1]t − 15 … t: receptive field 1 + (1+2+4+8) = 16 samplesWaveNet: d = 1, 2, …, 512 repeated ×3 → 1 + 3·1023 = 3070 samples (192 ms at 16 kHz)one residual layerresidual stream indilated conv k = 2dilation d, causalcondition hspeaker / meltanhfilterσgate⊙1×1 conv1×1 → skipskip s_lresidual+to layer l + 1z = tanh(W_f∗x + V_f·h) ⊙ σ(W_g∗x + V_g·h)Σ skipsReLU1×1ReLU1×1softmax 256sample x_t8-bit μ-law class (μ = 255), fed back as the next input sample
fig.Left: four causal layers with kernel 2 and dilations 1, 2, 4, 8. The highlighted tree shows that the last output depends on exactly the 16 most recent inputs,. Right: one residual layer. The dilated convolution feeds a tanh filter and a sigmoid gate, conditioning h is added to both, and their product goes through a 1×1 convolution into the residual stream and, separately, into a skip output. Bottom: skips from all layers are summed, passed through ReLU and 1×1 layers, and turned into a 256-way softmax. The sampled class is decoded and fed back as the next input.

Gates, residuals and skips

z=tanh⁡ ⁣(Wf∗x+Vf h)⊙σ ⁣(Wg∗x+Vg h)\mathbf z=\tanh\!\big(W_{f}\ast\mathbf x+V_{f}\,\mathbf h\big)\odot\sigma\!\big(W_{g}\ast\mathbf x+V_{g}\,\mathbf h\big)
Wf, WgW_f,\ W_g
dilated convolution filters for the filter and the gate branch of one layer; ∗\ast is convolution
⊙, σ\odot,\ \sigma
element-wise product and logistic sigmoid. This gated unit, taken from PixelCNN, worked better than ReLU for audio in the paper
h\mathbf h
conditioning. Global: one vector per utterance, such as a speaker embedding. Local: a time series such as linguistic features or a mel spectrogram, first upsampled to the audio rate (with a transposed convolution or by repetition), then added through a 1×1 convolution

Each layer's output z\mathbf z passes through a 1×1 convolution. The result is added to the layer input (a residual connection) and is also sent to a separate skip path. The output head sums the skips of all layers and applies ReLU, 1×1, ReLU, 1×1 and a softmax. Residual and skip paths let a stack of dozens of layers train, and let the head see features from every time scale.

μ-law: 256 classes instead of 65,536

WaveNet treats each sample as a classification target. A categorical softmax makes no assumption about the shape of the distribution, so it can be multi-modal. Raw 16-bit audio would need 65,536 classes per step, however. The paper first compresses amplitudes with μ-law companding and then quantises to 8 bits:

F(x)=sign⁡(x) ln⁡(1+μ∣x∣)ln⁡(1+μ),−1<x<1,μ=255F(x)=\operatorname{sign}(x)\,\frac{\ln(1+\mu|x|)}{\ln(1+\mu)},\qquad -1<x<1,\quad \mu=255
F(x)F(x)
compressed amplitude in (−1,1)(-1,1), then cut into 256 uniform bins. Back in xx, the bins next to zero are 0.00017 wide and those at full scale 0.043, about 245 times wider
μ\mu
compression strength; 255 is the value used by G.711 telephony and by WaveNet

Uniform 8-bit steps are 2/256=0.00782/256=0.0078 wide everywhere. Quiet passages, which are common in speech, would then use only a handful of levels. μ-law spends its bins where the samples are, so the relative error stays roughly constant across loudness. The authors report that the 8-bit μ-law reconstruction sounded very similar to the original.

8-bit quantisation (256 levels) of a 128 ms vowel at 16 kHz · peak level -12 dBFSwaveform (rescaled to fit) · 2048 samples · F0 = 125 Hzzoom ↓ quiet tailsamples 1700–1763 (4 ms) · original (dashed) · linear: 2.8 dB here · μ-law: 32.6 dB hereF(x) = sign(x)·ln(1+μ|x|)/ln(1+μ), μ = 255linμdecision thresholds in x: μ-law packs them near 0SNR (dB) vs peak level (dBFS)0204060-48-240whole clip: linear SNR 25.5 dB (56 of 256 levels used) · μ-law SNR 37.8 dB (185 used)
μ
fig.A synthetic 128 ms vowel (F0 = 125 Hz) quantised two ways. At the defaults (peak −12 dBFS, 8 bits, μ = 255) linear PCM gets an SNR of 25.5 dB over the clip and uses only 56 of its 256 levels. μ-law reaches 37.8 dB using 185 levels. In the zoomed quiet tail the gap is 2.8 dB against 32.6 dB: linear PCM turns the tail into a square wave. Drop the level to −30 dBFS: linear falls to 5.7 dB (9 levels), while μ-law holds 34.5 dB. At 0 dBFS the two nearly meet (37.5 vs 37.9 dB). The SNR curve shows linear losing about 1 dB per dB of level, while μ-law stays flat over a wide range.
xF(x)μ-law classdecoded|error| μ-law|error| linear
0.50.87572400.509030.009030.00391
−0.2−0.712636−0.202600.002600.00078
0.050.47271880.050000.000000.00078
0.010.22851570.010150.000150.00172
0.0010.04101330.001060.000060.00291
Worked example: 8-bit encoding of five sample values (μ = 255, mid-rise bins; computed with numpy)

Loud samples lose a little precision (the μ-law bin containing 0.5 is 0.022 wide), while quiet ones gain a lot (the bin at 0.001 is 0.00022 wide). At 0.001, linear 8-bit cannot tell the sample from zero: it lands in the first bin above zero, whose centre is 0.0039.

From the paper (van den Oord et al., 2016)

Dilations
1, 2, 4, …, 512, repeated (the paper's example: three times)
Output
256-way softmax over 8-bit μ-law values (μ = 255)
Multi-speaker
VCTK, 44 hours from 109 speakers, with a one-hot speaker ID as global conditioning
TTS data
single-speaker North American English (24.6 h) and Mandarin Chinese, conditioned on linguistic features plus log F0
US English MOS
WaveNet 4.21 vs 3.86 (HMM-driven unit selection) and 3.67 (LSTM parametric); natural 16-bit speech 4.55

Sampling is slow

Training is parallel, because every target xtx_t is known. Generation is not. Sample tt must be drawn and fed back before sample t+1t+1 can be computed, so one second of 16 kHz audio needs 16,000 sequential passes through the whole stack. Caching activations of earlier steps removes recomputation, but the chain of dependencies remains.

Parallel WaveNet (van den Oord et al., 2018) trained a non-autoregressive inverse-autoregressive-flow student to match a trained WaveNet teacher (probability density distillation, a form of Knowledge distillation). The student generates all samples at once and ran more than 20 times faster than real time at 24 kHz. WaveRNN attacked the same cost with a small recurrent network, and GAN vocoders such as HiFi-GAN removed autoregression altogether. WaveNet also became the vocoder of Tacotron 2, conditioned on mel spectrograms.

python
1import numpy as np
2
3def mu_law_encode(x, mu=255): # x in (-1, 1) -> integer class 0..255
4 y = np.sign(x) * np.log1p(mu * np.abs(x)) / np.log1p(mu)
5 return np.clip(((y + 1) / 2 * (mu + 1)).astype(int), 0, mu)
6
7def mu_law_decode(q, mu=255): # class -> bin centre, expanded back
8 y = 2 * (q + 0.5) / (mu + 1) - 1
9 return np.sign(y) * ((1 + mu) ** np.abs(y) - 1) / mu
10
11sr = 16000
12x = 0.3 * np.sin(2 * np.pi * 440 * np.arange(sr) / sr)
13q = mu_law_encode(x) # the training targets for the softmax
14xh = mu_law_decode(q)
15print(q.min(), q.max(), 10 * np.log10(np.sum(x**2) / np.sum((x - xh)**2))) # 27 228 38.1
16
17dilations = [2**i for i in range(10)] * 3 # 30 layers, kernel size 2
18R = 1 + sum(dilations)
19print(R, 1000 * R / sr, "ms") # 3070 191.875 ms
μ-law encode/decode and the receptive field of a WaveNet stack (numpy; runs as is)

Check yourself

A WaveNet uses kernel size 2 and dilations 1, 2, …, 512 repeated twice (20 layers). What is its receptive field at 16 kHz?

Check yourself

Why does WaveNet quantise with μ-law rather than uniformly to 8 bits?

Remember

  • ✓Autoregressive over raw samples: p(x)=∏tp(xt∣x<t,h)p(x)=\prod_t p(x_t\mid x_{<t},\mathbf h); training is parallel, sampling is sequential.
  • ✓Dilated causal convolutions, d=1,…,512d=1,\dots,512 repeated: receptive field 1+1023B1+1023B samples (3,070 for B=3B=3, 192 ms at 16 kHz).
  • ✓Gated unit tanh⁡(⋅)⊙σ(⋅)\tanh(\cdot)\odot\sigma(\cdot), with residual and skip connections; global (speaker) and local (features, mel) conditioning.
  • ✓8-bit μ-law (μ=255\mu=255) → a 256-way softmax that can be multi-modal.
  • ✓Sampling cost motivated Parallel WaveNet, WaveRNN and GAN vocoders such as HiFi-GAN.
← back to Deep Learning roadmap