WaveRNN

WaveRNN generates 16-bit audio with a single 896-unit GRU: each sample is split into coarse and fine bytes, predicted by two 256-way softmaxes. Weight sparsification and subscale batching then make it run in real time, even on a phone CPU.

7 min read799 wordsadvanced0 viewsupdated 2026-10-15

Definition

WaveRNN is an autoregressive waveform model built from a single recurrent layer with a dual softmax output. It was designed for fast sampling: few sequential operations per sample, sparse weights and several samples per step. Introduced by Kalchbrenner, Elsen, Simonyan, Noury, Casagrande, Lockhart, Stimberg, van den Oord, Dieleman and Kavukcuoglu (DeepMind), "Efficient Neural Audio Synthesis" (ICML 2018).

The task is the same as WaveNet's: model p(xt∣x<t,h)p(x_t\mid x_{<t},\mathbf h) for raw audio, usually conditioned on linguistic features or a mel spectrogram. WaveNet's quality came at a high sampling cost. Every sample required a pass through dozens of convolutional layers, one after the other, and 24 kHz audio needs 24,000 such passes per second.

The paper frames the cost as a product. Generation time is the number of samples times the per-sample latency, and that latency is set by the number of sequential operations, not by their total size. Each operation pays its compute time plus a fixed overhead such as a kernel launch or a memory read of the weights.

T(u)=∣u∣∑i=1N(c(opi)+d(opi))T(\mathbf u)=|\mathbf u|\sum_{i=1}^{N}\big(c(\mathrm{op}_i)+d(\mathrm{op}_i)\big)
∣u∣|\mathbf u|
number of samples in the utterance (24,000 per second at 24 kHz)
NN
number of sequential matrix–vector operations per sample: large for a deep WaveNet, a handful for one GRU layer
c, dc,\ d
compute time and fixed overhead of each operation. Sparsity reduces c, a single fused GPU kernel reduces d, and subscale generation reduces the effective |u|

One GRU layer, two softmaxes

A 16-bit sample has 65,536 possible values. WaveNet avoided a 65,536-way softmax with 8-bit μ-law. WaveRNN keeps all 16 bits but splits each offset sample ut=xt+32768u_t=x_t+32768 into a coarse byte ct=⌊ut/256⌋c_t=\lfloor u_t/256\rfloor and a fine byte ft=ut mod 256f_t=u_t \bmod 256. Each is predicted with its own 256-way softmax, and the fine byte is predicted conditioned on the coarse byte of the same step.

xt=[ct−1,ft−1,ct],ut=σ(Ruht−1+Iu⋆xt),rt=σ(Rrht−1+Ir⋆xt)et=tanh⁡ ⁣(rt⊙(Reht−1)+Ie⋆xt),ht=ut⊙ht−1+(1−ut)⊙et[yc,yf]=ht,P(ct)=softmax(O2 relu(O1yc)),P(ft)=softmax(O4 relu(O3yf))\begin{aligned}\mathbf x_t&=[c_{t-1},f_{t-1},c_t],\quad \mathbf u_t=\sigma(R_u\mathbf h_{t-1}+I^\star_u\mathbf x_t),\quad \mathbf r_t=\sigma(R_r\mathbf h_{t-1}+I^\star_r\mathbf x_t)\\ \mathbf e_t&=\tanh\!\big(\mathbf r_t\odot(R_e\mathbf h_{t-1})+I^\star_e\mathbf x_t\big),\qquad \mathbf h_t=\mathbf u_t\odot\mathbf h_{t-1}+(1-\mathbf u_t)\odot\mathbf e_t\\ [\mathbf y_c,\mathbf y_f]&=\mathbf h_t,\qquad P(c_t)=\mathrm{softmax}\big(O_2\,\mathrm{relu}(O_1\mathbf y_c)\big),\qquad P(f_t)=\mathrm{softmax}\big(O_4\,\mathrm{relu}(O_3\mathbf y_f)\big)\end{aligned}
ht\mathbf h_t
GRU state (896 units in WaveRNN-896), split into a coarse half yc\mathbf y_c and a fine half yf\mathbf y_f. See Gated recurrent unit (GRU)
I⋆I^\star
masked input matrices: the current coarse byte ctc_t only feeds the fine half, so yc\mathbf y_c never depends on the value it predicts
O1,…,O4O_1,\dots,O_4
small output layers, one ReLU pair per softmax

Sampling a step therefore has two parts. First compute the coarse half and sample ctc_t. Then feed ctc_t in and compute the fine half to sample ftf_t. The waveform value is xt=256 ct+ft−32768x_t=256\,c_t+f_t-32768. The recurrent matrices RR are shared, so this costs only slightly more than one GRU step.

one GRU step, dual softmaxu = x + 32768 = 25339 = 256·98 + 2510110001011111011coarse c = 98fine f = 251x_t = [c_t−1, f_t−1, c_t]c_t masked: reaches fine half onlyGRU, 896 unitsgates u_t, r_t, candidate e_th_t−1y_c: 448y_f: 448O1 · ReLU · O2O3 · ReLU · O4softmax 256softmax 256sample c_tsample f_t1st: y_c gives c_t (y_c never sees c_t)2nd: feed the sampled c_t, y_f gives f_tx_t = 256·c_t + f_t − 32768 (16 bits, 24 kHz)2 × 256 logits instead of 65,536subscale generationfold x into B interleaved sub-sequences (B = 4 here)sub 0sub 1sub 2sub 3048121620242815913172125292610141822263037111519232731cell = original sample index · done · this batch stepsub i runs F steps behind sub i−1 (F = 2 here),so B samples come out per step (paper: B = 16)weight sparsificationrecurrent matrix inblocks (e.g. 4×4);smallest blocks arezeroed graduallyduring traininglarge + sparse beatssmall + dense at thesame weight count
fig.Left: the 16-bit word u = 25339 splits into coarse byte 98 and fine byte 251. One 896-unit GRU reads the previous coarse and fine values plus the current coarse value, masked so that it reaches only the fine half. The two 448-unit halves each feed two small layers and a 256-way softmax. Right top: subscale generation with B = 4 (the paper uses 16). Sub-sequence i holds samples i, i + B, i + 2B and so on, and runs F steps behind sub-sequence i − 1, so one batched step (orange) emits B samples at once. Right bottom: a block-sparse recurrent matrix.

Why the split works

The coarse byte carries the waveform's shape, which a recurrent state can track. The fine byte is close to uniform noise, but its error costs only 1 part in 65,536. A wrong coarse byte by one step costs 256 units, or 0.78% of full scale. Splitting the bits puts the model's capacity where the errors are audible.

16-bit samples at 24 kHz, peak -6 dBFS · u = x + 32768 = 256·c + f10 ms window (240 samples) · x_t · coarse only: 256·c + 128 (SNR 39.3 dB over the clip)c_t (coarse, 0–255)f_t (fine, 0–255)sample t = 60: x = -7429 → u = 25339 = 256 × 98 + 2510110001011111011high byte → coarse softmaxlow byte → fine softmaxc off by 1 → error 256f off by 1 → error 1coarse: 126 values used · H = 6.67 bitsfine: H = 7.91 bits (uniform = 8)histograms over the full 100 ms clip (2400 samples), 256 bins eachoutput: one 16-bit softmax = 65,536 logits · dual softmax = 2 × 256 = 512 logits
overlay
fig.A synthetic 24 kHz, 16-bit vowel (F0 = 150 Hz). At the defaults (peak −6 dBFS, sample t = 60) the selected sample x = −7429 gives u = 25339 = 256 × 98 + 251. Over the 100 ms clip the coarse byte takes 126 distinct values with entropy 6.67 bits, while the fine byte has 7.91 bits, almost exactly uniform. Keeping only the coarse byte (fine set to its midpoint 128) still gives 39.3 dB SNR. At −30 dBFS the coarse byte uses just 9 values (2.75 bits) and coarse-only SNR drops to 15.6 dB, so quiet audio relies on the fine softmax. At 0 dBFS the coarse byte uses 238 values (7.61 bits, 45.4 dB). The fine entropy stays near 7.92 bits at every level.

Worked example: encoding and decoding one sample

  1. 1Signed sample x=−7429x=-7429. Offset: u=−7429+32768=25339u=-7429+32768=25339 (binary 01100010 11111011).
  2. 2Coarse c=⌊25339/256⌋=98c=\lfloor 25339/256\rfloor=98; fine f=25339−256⋅98=251f=25339-256\cdot98=251. These are the two class targets.
  3. 3Decode: 256⋅98+251−32768=−7429256\cdot98+251-32768=-7429, exact. A coarse-only estimate 256⋅98+128−32768=−7552256\cdot98+128-32768=-7552 is 123 units off.
  4. 4If the model samples c=99c=99 instead of 98, the value moves by 256 units (0.78% of full scale). If it samples f=250f=250, it moves by 1 unit (0.003%).
  5. 5Output size: a 16-bit softmax needs 65,536 logits per step; the dual softmax needs 2×256=5122\times256=512.

Sparsity and subscale

Weight sparsification. During training the smallest-magnitude weights are pruned on a gradual schedule until a target sparsity is reached (see Pruning). Pruning in small structured blocks, such as 4×4 or 16×1, keeps memory reads contiguous so the sparse matrix–vector products are actually fast. The key finding: at a fixed number of non-zero weights, a large sparse network beats a small dense one, and this held for sparsity levels above 96%. The small number of weights is what made real-time sampling on a mobile CPU possible.

Subscale generation. A sequence of length LL is folded into BB interleaved sub-sequences of length L/BL/B. Sub-sequence jj holds samples j,j+B,j+2B,…j, j+B, j+2B,\dots. They are generated in order, and each sample is conditioned on the past of its own sub-sequence and on the earlier sub-sequences, including a limited window of their future. Sub-sequence ii only needs to lag i−1i-1 by a fixed horizon FF. All BB sub-sequences can then run together as one batch: with B=16B=16, 16 samples come out per step instead of one.

From the paper (Kalchbrenner et al., 2018)

Audio
24 kHz, 16-bit, text-to-speech conditioned on linguistic features
WaveRNN-896
one GRU layer with 896 units and the dual softmax; reported to match WaveNet's quality
GPU speed
generates 24 kHz 16-bit audio about 4× faster than real time, using a custom kernel that runs the sampling loop as one persistent GPU operation
Sparse WaveRNN
large sparse nets beat small dense ones at equal weight count, beyond 96% sparsity; real time on a mobile CPU
Subscale WaveRNN
B = 16 sub-sequences generated together: 16 samples per step

WaveRNN became a common neural vocoder for mel-conditioned TTS. LPCNet (Valin and Skoglund, 2019) combined a WaveRNN-style network with classic linear prediction, so the network only models the prediction residual, and ran on ordinary CPUs. Non-autoregressive GAN vocoders such as HiFi-GAN later removed the per-sample loop entirely.

python
1import numpy as np
2
3sr = 24000
4t = np.arange(sr) / sr
5rng = np.random.default_rng(0)
6x = (0.5 * 32767 * np.sin(2 * np.pi * 150 * t) + rng.normal(0, 30, sr)).astype(np.int16) # stand-in for a 16-bit wav
7
8u = x.astype(np.int32) + 32768 # 0 .. 65535
9coarse, fine = u >> 8, u & 0xFF # two 256-class targets per sample
10assert np.all(256 * coarse + fine - 32768 == x)
11
12def entropy(v):
13 p = np.bincount(v, minlength=256) / len(v)
14 p = p[p > 0]
15 return -(p * np.log2(p)).sum()
16
17x_coarse = 256 * coarse + 128 - 32768 # what you get if only the coarse byte is right
18snr = 10 * np.log10(np.sum(x.astype(float)**2) / np.sum((x - x_coarse).astype(float)**2))
19print(f"H(coarse)={entropy(coarse):.2f} bits H(fine)={entropy(fine):.2f} bits coarse-only SNR={snr:.1f} dB")
20# H(coarse)=6.25 bits H(fine)=7.99 bits coarse-only SNR=43.8 dB
Coarse/fine targets from 16-bit audio, and what each byte is worth (numpy; runs as is)

Use when

  • +Full 16-bit output without a 65,536-way softmax or μ-law companding
  • +Very few sequential operations per sample; fast with fused kernels
  • +Sparsity and subscale bring real-time synthesis to mobile CPUs

Avoid when

  • −Still autoregressive: one (batched) recurrent step per sample or per B samples
  • −Custom sparse and persistent kernels are needed to realise the speed
  • −Later superseded for most uses by parallel GAN vocoders

Check yourself

A 16-bit sample has offset value u = 40000. What are its coarse and fine targets?

Check yourself

With subscale generation and B = 16, how many sequential batched steps does one second of 24 kHz audio need, ignoring the start-up stagger?

Remember

  • ✓Single 896-unit GRU; per-sample latency is set by the sequential operation count NN: T=∣u∣∑(c+d)T=|\mathbf u|\sum(c+d).
  • ✓Dual softmax: u=x+32768=256c+fu=x+32768=256c+f; predict ctc_t, then ftf_t given ctc_t (masked inputs).
  • ✓Coarse byte carries the shape; fine byte is near-uniform but costs only 1/65,536 per error.
  • ✓Block-sparse weights: large sparse beats small dense, even beyond 96% sparsity; real time on a mobile CPU.
  • ✓Subscale: fold the sequence into B = 16 interleaved parts and generate 16 samples per step.
← back to Deep Learning roadmap