# WaveNet

> WaveNet predicts raw audio one sample at a time: a 256-way softmax over μ-law classes (μ = 255), fed by stacks of gated, dilated causal convolutions (d = 1…512, repeated) whose receptive field grows exponentially with depth.

Source: https://www.mlfoundry.in/learn/wavenet  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**WaveNet:** is an autoregressive generative model of raw audio waveforms. It factorises $p(x)=\prod_t p(x_t\mid x_1,\dots,x_{t-1})$ and models each conditional with a deep stack of dilated causal convolutions ending in a softmax over 256 quantised amplitude values. Introduced by van den Oord, Dieleman, Zen, Simonyan, Vinyals, Graves, Kalchbrenner, Senior and Kavukcuoglu (DeepMind), "WaveNet: A Generative Model for Raw Audio" (2016).

The task: given past samples, and optionally a speaker or linguistic features, output the next sample. Audio is extremely dense. One second at 16 kHz is 16,000 values, and speech has structure at many time scales at once: pitch periods of a few milliseconds, phonemes of tens of milliseconds and words lasting hundreds of milliseconds.

Before WaveNet, speech synthesis either concatenated recorded units or ran a hand-designed signal-processing vocoder on predicted acoustic parameters (see **Text-to-speech**). Both were limited by the vocoder or by the unit database. WaveNet removed the vocoder: the network itself is the waveform model. It applies the idea of PixelCNN (see **Autoregressive generative models**) to one-dimensional sound.

## Dilated causal convolutions

A **causal** convolution only reads $x_{\le t}$, so the model can be trained in parallel with teacher forcing and still be sampled left to right. With kernel size 2 and no dilation, each layer adds only one sample of context, so covering 100 ms (1,600 samples at 16 kHz) would take about 1,600 layers. A **dilated** convolution (see **Dilated convolution**) skips $d-1$ inputs between taps. Doubling $d$ at every layer makes the receptive field grow exponentially with depth.

$$
R \;=\; 1+\sum_{l=1}^{L}(k-1)\,d_l,\qquad d_l\in\{1,2,4,\dots,512\}\ \text{repeated }B\text{ times}\;\Rightarrow\;R=1+1023\,B\ \ (k=2)
$$

- $R$: receptive field in samples: how far back the output at $t$ can see
- $k,\ d_l$: kernel size (2 in WaveNet) and dilation of layer $l$
- $B$: number of repeated 10-layer blocks. One block gives 1,024 samples (64 ms at 16 kHz); the paper's example of three blocks gives 3,070 samples (192 ms)

> Figure: Left: four causal layers with kernel 2 and dilations 1, 2, 4, 8. The highlighted tree shows that the last output depends on exactly the 16 most recent inputs,. Right: one residual layer. The dilated convolution feeds a tanh filter and a sigmoid gate, conditioning h is added to both, and their product goes through a 1×1 convolution into the residual stream and, separately, into a skip output. Bottom: skips from all layers are summed, passed through ReLU and 1×1 layers, and turned into a 256-way softmax. The sampled class is decoded and fed back as the next input.

## Gates, residuals and skips

$$
\mathbf z=\tanh\!\big(W_{f}\ast\mathbf x+V_{f}\,\mathbf h\big)\odot\sigma\!\big(W_{g}\ast\mathbf x+V_{g}\,\mathbf h\big)
$$

- $W_f,\ W_g$: dilated convolution filters for the filter and the gate branch of one layer; $\ast$ is convolution
- $\odot,\ \sigma$: element-wise product and logistic sigmoid. This gated unit, taken from PixelCNN, worked better than ReLU for audio in the paper
- $\mathbf h$: conditioning. **Global**: one vector per utterance, such as a speaker embedding. **Local**: a time series such as linguistic features or a mel spectrogram, first upsampled to the audio rate (with a transposed convolution or by repetition), then added through a 1×1 convolution

Each layer's output $\mathbf z$ passes through a 1×1 convolution. The result is added to the layer input (a residual connection) and is also sent to a separate skip path. The output head sums the skips of all layers and applies ReLU, 1×1, ReLU, 1×1 and a softmax. Residual and skip paths let a stack of dozens of layers train, and let the head see features from every time scale.

## μ-law: 256 classes instead of 65,536

WaveNet treats each sample as a **classification** target. A categorical softmax makes no assumption about the shape of the distribution, so it can be multi-modal. Raw 16-bit audio would need 65,536 classes per step, however. The paper first compresses amplitudes with μ-law companding and then quantises to 8 bits:

$$
F(x)=\operatorname{sign}(x)\,\frac{\ln(1+\mu|x|)}{\ln(1+\mu)},\qquad -1<x<1,\quad \mu=255
$$

- $F(x)$: compressed amplitude in $(-1,1)$, then cut into 256 uniform bins. Back in $x$, the bins next to zero are 0.00017 wide and those at full scale 0.043, about 245 times wider
- $\mu$: compression strength; 255 is the value used by G.711 telephony and by WaveNet

Uniform 8-bit steps are $2/256=0.0078$ wide everywhere. Quiet passages, which are common in speech, would then use only a handful of levels. μ-law spends its bins where the samples are, so the relative error stays roughly constant across loudness. The authors report that the 8-bit μ-law reconstruction sounded very similar to the original.

> Figure: A synthetic 128 ms vowel (F0 = 125 Hz) quantised two ways. At the defaults (peak −12 dBFS, 8 bits, μ = 255) linear PCM gets an SNR of 25.5 dB over the clip and uses only 56 of its 256 levels. μ-law reaches 37.8 dB using 185 levels. In the zoomed quiet tail the gap is 2.8 dB against 32.6 dB: linear PCM turns the tail into a square wave. Drop the level to −30 dBFS: linear falls to 5.7 dB (9 levels), while μ-law holds 34.5 dB. At 0 dBFS the two nearly meet (37.5 vs 37.9 dB). The SNR curve shows linear losing about 1 dB per dB of level, while μ-law stays flat over a wide range.

Worked example: 8-bit encoding of five sample values (μ = 255, mid-rise bins; computed with numpy)

| x | F(x) | μ-law class | decoded | \|error\| μ-law | \|error\| linear |
| --- | --- | --- | --- | --- | --- |
| 0.5 | 0.8757 | 240 | 0.50903 | 0.00903 | 0.00391 |
| −0.2 | −0.7126 | 36 | −0.20260 | 0.00260 | 0.00078 |
| 0.05 | 0.4727 | 188 | 0.05000 | 0.00000 | 0.00078 |
| 0.01 | 0.2285 | 157 | 0.01015 | 0.00015 | 0.00172 |
| 0.001 | 0.0410 | 133 | 0.00106 | 0.00006 | 0.00291 |

Loud samples lose a little precision (the μ-law bin containing 0.5 is 0.022 wide), while quiet ones gain a lot (the bin at 0.001 is 0.00022 wide). At 0.001, linear 8-bit cannot tell the sample from zero: it lands in the first bin above zero, whose centre is 0.0039.

**From the paper (van den Oord et al., 2016)**

- **Dilations:** 1, 2, 4, …, 512, repeated (the paper's example: three times)
- **Output:** 256-way softmax over 8-bit μ-law values (μ = 255)
- **Multi-speaker:** VCTK, 44 hours from 109 speakers, with a one-hot speaker ID as global conditioning
- **TTS data:** single-speaker North American English (24.6 h) and Mandarin Chinese, conditioned on linguistic features plus log F0
- **US English MOS:** WaveNet 4.21 vs 3.86 (HMM-driven unit selection) and 3.67 (LSTM parametric); natural 16-bit speech 4.55

## Sampling is slow

Training is parallel, because every target $x_t$ is known. Generation is not. Sample $t$ must be drawn and fed back before sample $t+1$ can be computed, so one second of 16 kHz audio needs 16,000 sequential passes through the whole stack. Caching activations of earlier steps removes recomputation, but the chain of dependencies remains.

**Parallel WaveNet** (van den Oord et al., 2018) trained a non-autoregressive inverse-autoregressive-flow student to match a trained WaveNet teacher (probability density distillation, a form of **Knowledge distillation**). The student generates all samples at once and ran more than 20 times faster than real time at 24 kHz. **WaveRNN** attacked the same cost with a small recurrent network, and GAN vocoders such as **HiFi-GAN** removed autoregression altogether. WaveNet also became the vocoder of **Tacotron 2**, conditioned on mel spectrograms.

```python
import numpy as np

def mu_law_encode(x, mu=255):                # x in (-1, 1) -> integer class 0..255
    y = np.sign(x) * np.log1p(mu * np.abs(x)) / np.log1p(mu)
    return np.clip(((y + 1) / 2 * (mu + 1)).astype(int), 0, mu)

def mu_law_decode(q, mu=255):                # class -> bin centre, expanded back
    y = 2 * (q + 0.5) / (mu + 1) - 1
    return np.sign(y) * ((1 + mu) ** np.abs(y) - 1) / mu

sr = 16000
x = 0.3 * np.sin(2 * np.pi * 440 * np.arange(sr) / sr)
q = mu_law_encode(x)                         # the training targets for the softmax
xh = mu_law_decode(q)
print(q.min(), q.max(), 10 * np.log10(np.sum(x**2) / np.sum((x - xh)**2)))   # 27 228 38.1

dilations = [2**i for i in range(10)] * 3    # 30 layers, kernel size 2
R = 1 + sum(dilations)
print(R, 1000 * R / sr, "ms")                # 3070 191.875 ms
```

μ-law encode/decode and the receptive field of a WaveNet stack (numpy; runs as is)

> **Why classification beats regression here:** A Gaussian or L2 output predicts the mean of the next sample. When the next sample could plausibly be either positive or negative, the mean is near zero and the audio comes out muffled. A 256-way softmax can put mass on both modes, and sampling from it keeps the waveform sharp. Parallel WaveNet's teacher later moved to 16-bit output with a discretised mixture of logistics, which is still multi-modal.

> **Receptive field is not long-range memory:** Even 3,070 samples is only 192 ms at 16 kHz. The paper's VCTK model, conditioned only on speaker identity, produced word-like sounds but no coherent sentences, and the authors blamed the limited receptive field (about 300 ms). Prosody and content over seconds must come from the conditioning features $\mathbf h$ (linguistic features or a mel spectrogram), not from the waveform context.

**Question:** A WaveNet uses kernel size 2 and dilations 1, 2, …, 512 repeated twice (20 layers). What is its receptive field at 16 kHz?

- A. 20 samples (1.25 ms)
- B. 1,024 samples (64 ms)
- C. 2,047 samples (about 128 ms)
- D. 1,048,576 samples (65 s)

**Answer:** C. 2,047 samples (about 128 ms) — $R=1+2\times(1+2+\dots+512)=1+2\times1023=2047$ samples, and $2047/16000\approx0.128$ s. Without dilation the same 20 layers would see only 21 samples.

**Question:** Why does WaveNet quantise with μ-law rather than uniformly to 8 bits?

- A. μ-law makes the softmax layer smaller than 256 outputs
- B. Uniform bins waste resolution on rare loud values; μ-law bins are narrow near zero, where most speech samples lie, so quiet sounds keep their detail
- C. μ-law removes the need for causal convolutions
- D. Uniform quantisation cannot represent negative values

**Answer:** B. Uniform bins waste resolution on rare loud values; μ-law bins are narrow near zero, where most speech samples lie, so quiet sounds keep their detail — Both give 256 classes. μ-law's logarithmic spacing keeps the relative error roughly constant, so quiet passages are reproduced far better (in the figure: 34.5 dB vs 5.7 dB SNR at −30 dBFS).

**Key points**

- Autoregressive over raw samples: $p(x)=\prod_t p(x_t\mid x_{<t},\mathbf h)$; training is parallel, sampling is sequential.
- Dilated causal convolutions, $d=1,\dots,512$ repeated: receptive field $1+1023B$ samples (3,070 for $B=3$, 192 ms at 16 kHz).
- Gated unit $\tanh(\cdot)\odot\sigma(\cdot)$, with residual and skip connections; global (speaker) and local (features, mel) conditioning.
- 8-bit μ-law ($\mu=255$) → a 256-way softmax that can be multi-modal.
- Sampling cost motivated Parallel WaveNet, WaveRNN and GAN vocoders such as HiFi-GAN.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
