Definition
HiFi-GAN is a non-autoregressive neural vocoder trained as a Generative adversarial network (GAN). A fully convolutional generator maps a mel spectrogram to a waveform in one forward pass. Two families of discriminators, multi-period and multi-scale, judge the result. Introduced by Jungil Kong, Jaehyeon Kim and Jaekyoung Bae (Kakao Enterprise), "HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis" (NeurIPS 2020).
A vocoder turns the acoustic model's mel spectrogram into audio (see Text-to-speech for the pipeline). At 22,050 Hz with a 256-sample hop, every mel frame must become 256 samples. The mel throws away phase, so the vocoder has to invent a phase that sounds natural. Errors in periodic structure (pitch harmonics) are especially audible as buzz or metallic artefacts.
Autoregressive vocoders such as WaveNet and WaveRNN sound good but generate one sample at a time. Flow-based WaveGlow is parallel but large. Earlier GAN vocoders such as MelGAN (Kumar et al., 2019) were fast, but their quality lagged behind. HiFi-GAN's argument is that the discriminator must look explicitly at periodic patterns, because speech is a sum of sinusoids with different periods.
Generator: upsampling with multi-receptive-field fusion
The generator starts with a 7-tap convolution to channels (512 in V1). Four transposed-convolution stages (see Transposed convolution) then upsample by 8, 8, 2 and 2, a total of 256, the hop length. Each stage halves the channels. After each upsampling comes a multi-receptive-field fusion (MRF) module: residual blocks with kernel sizes 3, 7 and 11 and dilations 1, 3 and 5 (see Dilated convolution) run in parallel, and their outputs are averaged. Short kernels capture fine detail and long dilated ones capture longer patterns, all at the same resolution. A final convolution and tanh give the waveform.
Discriminators
Multi-period discriminator (MPD). It has five sub-discriminators, one per period . Each reflect-pads the waveform to a multiple of and reshapes it from length to a 2-D map of height and width . Its 2-D convolutions have kernels of width 1 along the period axis, (5, 1), with stride (3, 1), so each column is processed independently. A column holds samples . The sub-discriminator therefore sees the signal at every phase of period . The periods are primes, to avoid overlap between what the sub-discriminators see.
- 2-D input of the period- sub-discriminator. Moving one step down a column moves samples forward in time
- waveform padded on the right by reflection so its length is a multiple of
Multi-scale discriminator (MSD), taken from MelGAN, has three 1-D sub-discriminators. They see the raw waveform and versions average-pooled by 2 and by 4. These catch consecutive patterns and long-term structure, which the MPD, reading isolated sample columns, can miss. The raw-scale one uses spectral normalisation and the others weight normalisation.
Losses
- the 8 sub-discriminators (5 MPD + 3 MSD), trained with the least-squares GAN objective (see Adversarial loss)
- feature map of layer of sub-discriminator , and its number of elements: the feature-matching loss
- log-mel spectrogram (n_fft 1024, hop 256, 80 bands), so the mel loss is an L1 reconstruction term on what the acoustic model predicted
- 2 and 45
The mel loss anchors the generator to the input from the first step and stabilises early training. The adversarial and feature-matching terms then add the fine detail that an L1 loss on spectra would blur, especially phase and harmonic structure.
| Version | h_u | Upsampling | MRF kernels | Params | MOS |
|---|---|---|---|---|---|
| V1 | 512 | 8, 8, 2, 2 | 3, 7, 11 | 13.92M | 4.36 |
| V2 | 128 | 8, 8, 2, 2 | 3, 7, 11 | 0.92M | 4.23 |
| V3 | 256 | 8, 8, 4 | 3, 5, 7 (lighter blocks) | 1.46M | 4.05 |
Setting
- Data
- LJSpeech: 13,100 clips, about 24 hours, one female speaker, 22,050 Hz
- Baselines (MOS)
- WaveNet (mixture of logistics) 4.02, WaveGlow 3.81, MelGAN 3.79
- Training
- random 8192-sample segments; AdamW with β₁ = 0.8, β₂ = 0.99, learning rate 2 × 10⁻⁴ decayed by 0.999 per epoch
- Speed
- all versions run far faster than real time on a GPU; the small V2 and V3 also run faster than real time on a CPU
- Ablation
- removing the MPD caused the largest quality drop of any component
| p | 8192 mod p | reflect pad | 2-D shape (T/p, p) |
|---|---|---|---|
| 2 | 0 | 0 | 4096 × 2 |
| 3 | 2 | 1 | 2731 × 3 |
| 5 | 2 | 3 | 1639 × 5 |
| 7 | 2 | 5 | 1171 × 7 |
| 11 | 8 | 3 | 745 × 11 |
On the generator side, the same segment is mel frames, and the stages run samples. One second of audio is frames.
1import numpy as np23def fold(x, p):4 """MPD input: reflect-pad a 1-D waveform to a multiple of p, view as (T/p, p)."""5 T = len(x)6 if T % p:7 x = np.pad(x, (0, p - T % p), mode="reflect") # same as F.pad(..., "reflect") in PyTorch8 return x.reshape(-1, p) # row i holds x[i*p : i*p + p]910sr, f0 = 22050, 220.5 # a 100-sample pitch period11x = np.sin(2 * np.pi * f0 * np.arange(8192) / sr) # one 8192-sample training segment12for p in [2, 3, 5, 7, 11]:13 X = fold(x, p)14 # a (5, 1) kernel sees X[i:i+5, j] = x[t], x[t+p], ..., x[t+4p]15 r = np.corrcoef(X[:-1].ravel(), X[1:].ravel())[0, 1] # neighbours along the kernel axis16 print(p, X.shape, f"corr(x[t], x[t+p]) = {r:.3f}")17# 2 (4096, 2) 0.992 · 3 (2731, 3) 0.982 · 5 (1639, 5) 0.951 · 7 (1171, 7) 0.905 · 11 (745, 11) 0.771Use when
- +One parallel forward pass: orders of magnitude faster than autoregressive vocoders
- +V1 came close to ground truth (4.36 vs 4.45 MOS) on LJSpeech
- +V2 has under 1M parameters, practical on CPU and edge devices
Avoid when
- −GAN training: 8 sub-discriminators and many steps, sensitive to hyperparameters
- −Tied to its mel configuration; a new sample rate or hop needs retraining
- −Quality drops on mismatched predicted mels unless fine-tuned
Check yourself
A 2000-sample waveform enters the period-7 sub-discriminator. What 2-D shape does it get?
Check yourself
What does the multi-scale discriminator see that the multi-period discriminator does not?
Remember
- ✓Generator: conv → transposed convs ×8, ×8, ×2, ×2 (= hop 256) each followed by MRF (kernels 3, 7, 11; dilations 1, 3, 5) → tanh.
- ✓MPD: reflect-pad, fold for ; (5, 1) kernels compare samples apart.
- ✓MSD (from MelGAN): raw, 2× and 4× average-pooled waveforms.
- ✓ = LSGAN adversarial + 2 · feature matching + 45 · mel L1.
- ✓LJSpeech MOS: V1 4.36 (13.92M params), V2 4.23 (0.92M), V3 4.05; ground truth 4.45.