# WaveRNN

> WaveRNN generates 16-bit audio with a single 896-unit GRU: each sample is split into coarse and fine bytes, predicted by two 256-way softmaxes. Weight sparsification and subscale batching then make it run in real time, even on a phone CPU.

Source: https://www.mlfoundry.in/learn/wavernn  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 7 min read · updated 2026-10-15

**WaveRNN:** is an autoregressive waveform model built from a single recurrent layer with a **dual softmax** output. It was designed for fast sampling: few sequential operations per sample, sparse weights and several samples per step. Introduced by Kalchbrenner, Elsen, Simonyan, Noury, Casagrande, Lockhart, Stimberg, van den Oord, Dieleman and Kavukcuoglu (DeepMind), "Efficient Neural Audio Synthesis" (ICML 2018).

The task is the same as **WaveNet**'s: model $p(x_t\mid x_{<t},\mathbf h)$ for raw audio, usually conditioned on linguistic features or a mel spectrogram. WaveNet's quality came at a high sampling cost. Every sample required a pass through dozens of convolutional layers, one after the other, and 24 kHz audio needs 24,000 such passes per second.

The paper frames the cost as a product. Generation time is the number of samples times the per-sample latency, and that latency is set by the number of **sequential** operations, not by their total size. Each operation pays its compute time plus a fixed overhead such as a kernel launch or a memory read of the weights.

$$
T(\mathbf u)=|\mathbf u|\sum_{i=1}^{N}\big(c(\mathrm{op}_i)+d(\mathrm{op}_i)\big)
$$

- $|\mathbf u|$: number of samples in the utterance (24,000 per second at 24 kHz)
- $N$: number of sequential matrix–vector operations per sample: large for a deep WaveNet, a handful for one GRU layer
- $c,\ d$: compute time and fixed overhead of each operation. Sparsity reduces c, a single fused GPU kernel reduces d, and subscale generation reduces the effective |u|

## One GRU layer, two softmaxes

A 16-bit sample has 65,536 possible values. WaveNet avoided a 65,536-way softmax with 8-bit μ-law. WaveRNN keeps all 16 bits but splits each offset sample $u_t=x_t+32768$ into a **coarse** byte $c_t=\lfloor u_t/256\rfloor$ and a **fine** byte $f_t=u_t \bmod 256$. Each is predicted with its own 256-way softmax, and the fine byte is predicted conditioned on the coarse byte of the same step.

$$
\begin{aligned}\mathbf x_t&=[c_{t-1},f_{t-1},c_t],\quad \mathbf u_t=\sigma(R_u\mathbf h_{t-1}+I^\star_u\mathbf x_t),\quad \mathbf r_t=\sigma(R_r\mathbf h_{t-1}+I^\star_r\mathbf x_t)\\ \mathbf e_t&=\tanh\!\big(\mathbf r_t\odot(R_e\mathbf h_{t-1})+I^\star_e\mathbf x_t\big),\qquad \mathbf h_t=\mathbf u_t\odot\mathbf h_{t-1}+(1-\mathbf u_t)\odot\mathbf e_t\\ [\mathbf y_c,\mathbf y_f]&=\mathbf h_t,\qquad P(c_t)=\mathrm{softmax}\big(O_2\,\mathrm{relu}(O_1\mathbf y_c)\big),\qquad P(f_t)=\mathrm{softmax}\big(O_4\,\mathrm{relu}(O_3\mathbf y_f)\big)\end{aligned}
$$

- $\mathbf h_t$: GRU state (896 units in WaveRNN-896), split into a coarse half $\mathbf y_c$ and a fine half $\mathbf y_f$. See **Gated recurrent unit (GRU)**
- $I^\star$: masked input matrices: the current coarse byte $c_t$ only feeds the fine half, so $\mathbf y_c$ never depends on the value it predicts
- $O_1,\dots,O_4$: small output layers, one ReLU pair per softmax

Sampling a step therefore has two parts. First compute the coarse half and sample $c_t$. Then feed $c_t$ in and compute the fine half to sample $f_t$. The waveform value is $x_t=256\,c_t+f_t-32768$. The recurrent matrices $R$ are shared, so this costs only slightly more than one GRU step.

> Figure: Left: the 16-bit word u = 25339 splits into coarse byte 98 and fine byte 251. One 896-unit GRU reads the previous coarse and fine values plus the current coarse value, masked so that it reaches only the fine half. The two 448-unit halves each feed two small layers and a 256-way softmax. Right top: subscale generation with B = 4 (the paper uses 16). Sub-sequence i holds samples i, i + B, i + 2B and so on, and runs F steps behind sub-sequence i − 1, so one batched step (orange) emits B samples at once. Right bottom: a block-sparse recurrent matrix.

## Why the split works

The coarse byte carries the waveform's shape, which a recurrent state can track. The fine byte is close to uniform noise, but its error costs only 1 part in 65,536. A wrong coarse byte by one step costs 256 units, or 0.78% of full scale. Splitting the bits puts the model's capacity where the errors are audible.

> Figure: A synthetic 24 kHz, 16-bit vowel (F0 = 150 Hz). At the defaults (peak −6 dBFS, sample t = 60) the selected sample x = −7429 gives u = 25339 = 256 × 98 + 251. Over the 100 ms clip the coarse byte takes 126 distinct values with entropy 6.67 bits, while the fine byte has 7.91 bits, almost exactly uniform. Keeping only the coarse byte (fine set to its midpoint 128) still gives 39.3 dB SNR. At −30 dBFS the coarse byte uses just 9 values (2.75 bits) and coarse-only SNR drops to 15.6 dB, so quiet audio relies on the fine softmax. At 0 dBFS the coarse byte uses 238 values (7.61 bits, 45.4 dB). The fine entropy stays near 7.92 bits at every level.

**Worked example: encoding and decoding one sample**

1. Signed sample $x=-7429$. Offset: $u=-7429+32768=25339$ (binary 01100010 11111011).
2. Coarse $c=\lfloor 25339/256\rfloor=98$; fine $f=25339-256\cdot98=251$. These are the two class targets.
3. Decode: $256\cdot98+251-32768=-7429$, exact. A coarse-only estimate $256\cdot98+128-32768=-7552$ is 123 units off.
4. If the model samples $c=99$ instead of 98, the value moves by 256 units (0.78% of full scale). If it samples $f=250$, it moves by 1 unit (0.003%).
5. Output size: a 16-bit softmax needs 65,536 logits per step; the dual softmax needs $2\times256=512$.

## Sparsity and subscale

**Weight sparsification.** During training the smallest-magnitude weights are pruned on a gradual schedule until a target sparsity is reached (see **Pruning**). Pruning in small structured blocks, such as 4×4 or 16×1, keeps memory reads contiguous so the sparse matrix–vector products are actually fast. The key finding: at a fixed number of non-zero weights, a large sparse network beats a small dense one, and this held for sparsity levels above 96%. The small number of weights is what made real-time sampling on a mobile CPU possible.

**Subscale generation.** A sequence of length $L$ is folded into $B$ interleaved sub-sequences of length $L/B$. Sub-sequence $j$ holds samples $j, j+B, j+2B,\dots$. They are generated in order, and each sample is conditioned on the past of its own sub-sequence and on the earlier sub-sequences, including a limited window of their future. Sub-sequence $i$ only needs to lag $i-1$ by a fixed horizon $F$. All $B$ sub-sequences can then run together as one batch: with $B=16$, 16 samples come out per step instead of one.

**From the paper (Kalchbrenner et al., 2018)**

- **Audio:** 24 kHz, 16-bit, text-to-speech conditioned on linguistic features
- **WaveRNN-896:** one GRU layer with 896 units and the dual softmax; reported to match WaveNet's quality
- **GPU speed:** generates 24 kHz 16-bit audio about 4× faster than real time, using a custom kernel that runs the sampling loop as one persistent GPU operation
- **Sparse WaveRNN:** large sparse nets beat small dense ones at equal weight count, beyond 96% sparsity; real time on a mobile CPU
- **Subscale WaveRNN:** B = 16 sub-sequences generated together: 16 samples per step

WaveRNN became a common neural vocoder for mel-conditioned TTS. **LPCNet** (Valin and Skoglund, 2019) combined a WaveRNN-style network with classic linear prediction, so the network only models the prediction residual, and ran on ordinary CPUs. Non-autoregressive GAN vocoders such as **HiFi-GAN** later removed the per-sample loop entirely.

```python
import numpy as np

sr = 24000
t = np.arange(sr) / sr
rng = np.random.default_rng(0)
x = (0.5 * 32767 * np.sin(2 * np.pi * 150 * t) + rng.normal(0, 30, sr)).astype(np.int16)  # stand-in for a 16-bit wav

u = x.astype(np.int32) + 32768          # 0 .. 65535
coarse, fine = u >> 8, u & 0xFF         # two 256-class targets per sample
assert np.all(256 * coarse + fine - 32768 == x)

def entropy(v):
    p = np.bincount(v, minlength=256) / len(v)
    p = p[p > 0]
    return -(p * np.log2(p)).sum()

x_coarse = 256 * coarse + 128 - 32768   # what you get if only the coarse byte is right
snr = 10 * np.log10(np.sum(x.astype(float)**2) / np.sum((x - x_coarse).astype(float)**2))
print(f"H(coarse)={entropy(coarse):.2f} bits  H(fine)={entropy(fine):.2f} bits  coarse-only SNR={snr:.1f} dB")
# H(coarse)=6.25 bits  H(fine)=7.99 bits  coarse-only SNR=43.8 dB
```

Coarse/fine targets from 16-bit audio, and what each byte is worth (numpy; runs as is)

> **Latency, not FLOPs, limits autoregressive sampling:** At batch size 1, a matrix–vector product is bound by memory bandwidth and launch overhead, not arithmetic. Fewer, larger sequential operations (one GRU instead of a deep stack), fewer weights to read (sparsity) and more samples per sequential step (subscale) all shrink the per-sample latency. Total FLOPs matter far less.

> **The coarse byte must not see itself:** If $c_t$ leaks into the coarse half through an unmasked input matrix, teacher-forced training becomes trivial: the model copies its own target and the loss collapses. At sampling time $c_t$ is not yet known and the model fails. The mask on $I^\star$ is what keeps the factorisation $p(c_t)\,p(f_t\mid c_t)$ honest.

**Strengths**

- Full 16-bit output without a 65,536-way softmax or μ-law companding
- Very few sequential operations per sample; fast with fused kernels
- Sparsity and subscale bring real-time synthesis to mobile CPUs

**Limitations**

- Still autoregressive: one (batched) recurrent step per sample or per B samples
- Custom sparse and persistent kernels are needed to realise the speed
- Later superseded for most uses by parallel GAN vocoders

**Question:** A 16-bit sample has offset value u = 40000. What are its coarse and fine targets?

- A. c = 156, f = 64
- B. c = 64, f = 156
- C. c = 40, f = 0
- D. c = 157, f = 192

**Answer:** A. c = 156, f = 64 — $\lfloor 40000/256\rfloor=156$ and $40000-156\cdot256=40000-39936=64$.

**Question:** With subscale generation and B = 16, how many sequential batched steps does one second of 24 kHz audio need, ignoring the start-up stagger?

- A. 24,000
- B. 16
- C. 1,500
- D. 384,000

**Answer:** C. 1,500 — Each sub-sequence has $24000/16=1500$ samples, and all 16 advance together, one sample each per step. The stagger of $F$ steps per sub-sequence adds a fixed start-up cost.

**Key points**

- Single 896-unit GRU; per-sample latency is set by the sequential operation count $N$: $T=|\mathbf u|\sum(c+d)$.
- Dual softmax: $u=x+32768=256c+f$; predict $c_t$, then $f_t$ given $c_t$ (masked inputs).
- Coarse byte carries the shape; fine byte is near-uniform but costs only 1/65,536 per error.
- Block-sparse weights: large sparse beats small dense, even beyond 96% sparsity; real time on a mobile CPU.
- Subscale: fold the sequence into B = 16 interleaved parts and generate 16 samples per step.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
