Definition
WaveRNN is an autoregressive waveform model built from a single recurrent layer with a dual softmax output. It was designed for fast sampling: few sequential operations per sample, sparse weights and several samples per step. Introduced by Kalchbrenner, Elsen, Simonyan, Noury, Casagrande, Lockhart, Stimberg, van den Oord, Dieleman and Kavukcuoglu (DeepMind), "Efficient Neural Audio Synthesis" (ICML 2018).
The task is the same as WaveNet's: model for raw audio, usually conditioned on linguistic features or a mel spectrogram. WaveNet's quality came at a high sampling cost. Every sample required a pass through dozens of convolutional layers, one after the other, and 24 kHz audio needs 24,000 such passes per second.
The paper frames the cost as a product. Generation time is the number of samples times the per-sample latency, and that latency is set by the number of sequential operations, not by their total size. Each operation pays its compute time plus a fixed overhead such as a kernel launch or a memory read of the weights.
- number of samples in the utterance (24,000 per second at 24 kHz)
- number of sequential matrix–vector operations per sample: large for a deep WaveNet, a handful for one GRU layer
- compute time and fixed overhead of each operation. Sparsity reduces c, a single fused GPU kernel reduces d, and subscale generation reduces the effective |u|
One GRU layer, two softmaxes
A 16-bit sample has 65,536 possible values. WaveNet avoided a 65,536-way softmax with 8-bit μ-law. WaveRNN keeps all 16 bits but splits each offset sample into a coarse byte and a fine byte . Each is predicted with its own 256-way softmax, and the fine byte is predicted conditioned on the coarse byte of the same step.
- GRU state (896 units in WaveRNN-896), split into a coarse half and a fine half . See Gated recurrent unit (GRU)
- masked input matrices: the current coarse byte only feeds the fine half, so never depends on the value it predicts
- small output layers, one ReLU pair per softmax
Sampling a step therefore has two parts. First compute the coarse half and sample . Then feed in and compute the fine half to sample . The waveform value is . The recurrent matrices are shared, so this costs only slightly more than one GRU step.
Why the split works
The coarse byte carries the waveform's shape, which a recurrent state can track. The fine byte is close to uniform noise, but its error costs only 1 part in 65,536. A wrong coarse byte by one step costs 256 units, or 0.78% of full scale. Splitting the bits puts the model's capacity where the errors are audible.
Worked example: encoding and decoding one sample
- 1Signed sample . Offset: (binary 01100010 11111011).
- 2Coarse ; fine . These are the two class targets.
- 3Decode: , exact. A coarse-only estimate is 123 units off.
- 4If the model samples instead of 98, the value moves by 256 units (0.78% of full scale). If it samples , it moves by 1 unit (0.003%).
- 5Output size: a 16-bit softmax needs 65,536 logits per step; the dual softmax needs .
Sparsity and subscale
Weight sparsification. During training the smallest-magnitude weights are pruned on a gradual schedule until a target sparsity is reached (see Pruning). Pruning in small structured blocks, such as 4×4 or 16×1, keeps memory reads contiguous so the sparse matrix–vector products are actually fast. The key finding: at a fixed number of non-zero weights, a large sparse network beats a small dense one, and this held for sparsity levels above 96%. The small number of weights is what made real-time sampling on a mobile CPU possible.
Subscale generation. A sequence of length is folded into interleaved sub-sequences of length . Sub-sequence holds samples . They are generated in order, and each sample is conditioned on the past of its own sub-sequence and on the earlier sub-sequences, including a limited window of their future. Sub-sequence only needs to lag by a fixed horizon . All sub-sequences can then run together as one batch: with , 16 samples come out per step instead of one.
From the paper (Kalchbrenner et al., 2018)
- Audio
- 24 kHz, 16-bit, text-to-speech conditioned on linguistic features
- WaveRNN-896
- one GRU layer with 896 units and the dual softmax; reported to match WaveNet's quality
- GPU speed
- generates 24 kHz 16-bit audio about 4× faster than real time, using a custom kernel that runs the sampling loop as one persistent GPU operation
- Sparse WaveRNN
- large sparse nets beat small dense ones at equal weight count, beyond 96% sparsity; real time on a mobile CPU
- Subscale WaveRNN
- B = 16 sub-sequences generated together: 16 samples per step
WaveRNN became a common neural vocoder for mel-conditioned TTS. LPCNet (Valin and Skoglund, 2019) combined a WaveRNN-style network with classic linear prediction, so the network only models the prediction residual, and ran on ordinary CPUs. Non-autoregressive GAN vocoders such as HiFi-GAN later removed the per-sample loop entirely.
1import numpy as np23sr = 240004t = np.arange(sr) / sr5rng = np.random.default_rng(0)6x = (0.5 * 32767 * np.sin(2 * np.pi * 150 * t) + rng.normal(0, 30, sr)).astype(np.int16) # stand-in for a 16-bit wav78u = x.astype(np.int32) + 32768 # 0 .. 655359coarse, fine = u >> 8, u & 0xFF # two 256-class targets per sample10assert np.all(256 * coarse + fine - 32768 == x)1112def entropy(v):13 p = np.bincount(v, minlength=256) / len(v)14 p = p[p > 0]15 return -(p * np.log2(p)).sum()1617x_coarse = 256 * coarse + 128 - 32768 # what you get if only the coarse byte is right18snr = 10 * np.log10(np.sum(x.astype(float)**2) / np.sum((x - x_coarse).astype(float)**2))19print(f"H(coarse)={entropy(coarse):.2f} bits H(fine)={entropy(fine):.2f} bits coarse-only SNR={snr:.1f} dB")20# H(coarse)=6.25 bits H(fine)=7.99 bits coarse-only SNR=43.8 dBUse when
- +Full 16-bit output without a 65,536-way softmax or μ-law companding
- +Very few sequential operations per sample; fast with fused kernels
- +Sparsity and subscale bring real-time synthesis to mobile CPUs
Avoid when
- −Still autoregressive: one (batched) recurrent step per sample or per B samples
- −Custom sparse and persistent kernels are needed to realise the speed
- −Later superseded for most uses by parallel GAN vocoders
Check yourself
A 16-bit sample has offset value u = 40000. What are its coarse and fine targets?
Check yourself
With subscale generation and B = 16, how many sequential batched steps does one second of 24 kHz audio need, ignoring the start-up stagger?
Remember
- ✓Single 896-unit GRU; per-sample latency is set by the sequential operation count : .
- ✓Dual softmax: ; predict , then given (masked inputs).
- ✓Coarse byte carries the shape; fine byte is near-uniform but costs only 1/65,536 per error.
- ✓Block-sparse weights: large sparse beats small dense, even beyond 96% sparsity; real time on a mobile CPU.
- ✓Subscale: fold the sequence into B = 16 interleaved parts and generate 16 samples per step.