HiFi-GAN

HiFi-GAN is a GAN vocoder: transposed convolutions with multi-receptive-field fusion upsample a mel spectrogram 256×, judged by period discriminators (p = 2, 3, 5, 7, 11) and multi-scale discriminators, with a mel L1 loss weighted 45.

8 min read690 wordscore0 viewsupdated 2026-10-15

Definition

HiFi-GAN is a non-autoregressive neural vocoder trained as a Generative adversarial network (GAN). A fully convolutional generator maps a mel spectrogram to a waveform in one forward pass. Two families of discriminators, multi-period and multi-scale, judge the result. Introduced by Jungil Kong, Jaehyeon Kim and Jaekyoung Bae (Kakao Enterprise), "HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis" (NeurIPS 2020).

A vocoder turns the acoustic model's mel spectrogram into audio (see Text-to-speech for the pipeline). At 22,050 Hz with a 256-sample hop, every mel frame must become 256 samples. The mel throws away phase, so the vocoder has to invent a phase that sounds natural. Errors in periodic structure (pitch harmonics) are especially audible as buzz or metallic artefacts.

Autoregressive vocoders such as WaveNet and WaveRNN sound good but generate one sample at a time. Flow-based WaveGlow is parallel but large. Earlier GAN vocoders such as MelGAN (Kumar et al., 2019) were fast, but their quality lagged behind. HiFi-GAN's argument is that the discriminator must look explicitly at periodic patterns, because speech is a sum of sinusoids with different periods.

Generator: upsampling with multi-receptive-field fusion

The generator starts with a 7-tap convolution to huh_u channels (512 in V1). Four transposed-convolution stages (see Transposed convolution) then upsample by 8, 8, 2 and 2, a total of 256, the hop length. Each stage halves the channels. After each upsampling comes a multi-receptive-field fusion (MRF) module: residual blocks with kernel sizes 3, 7 and 11 and dilations 1, 3 and 5 (see Dilated convolution) run in parallel, and their outputs are averaged. Short kernels capture fine detail and long dilated ones capture longer patterns, all at the same resolution. A final convolution and tanh give the waveform.

generator G (V1): mel → waveform, 8 · 8 · 2 · 2 = 256 = hop length, channels halve each stagemel input80 × 32conv k7512 × 32↑8 + MRF256 × 256↑8 + MRF128 × 2048↑2 + MRF64 × 4096↑2 + MRF32 × 8192conv k7·tanh1 × 8192↑r = transposed conv (kernel 2r, stride r) + leaky ReLU, then MRFmulti-receptive-field fusion (MRF)inC chResBlock k = 3dilations 1, 3, 5ResBlock k = 7dilations 1, 3, 5ResBlock k = 11dilations 1, 3, 5+÷3ResBlock k, for d = 1, 3, 5 in turn:x ← x + conv(k, 1)(LReLU(conv(k, d)(LReLU(x))))MPD: fold 1-D audio by period p01234567891011x, T = 12view (T/p, p) with p = 301234567891011a (5, 1) kernel runs downeach column: samples p apartpad T to a multiple of p(reflect) before foldingreal x8192 samplesG(s)generatedp=22-Dp=32-Dp=52-Dp=72-Dp=112-DMPD: 5 period sub-discriminators, Conv2d (5, 1)raw audiospectral normpool ×2weight normpool ×4weight normMSD (from MelGAN): 3 scales, 1-D grouped convseach sub-discriminator outputsa score map + feature mapsL_D = (D(x) − 1)² + D(G(s))²summed over all 8 sub-DsL_G = Σ (D(G(s)) − 1)² + 2 · L_FM + 45 · L_melL_FM: L1 between D feature maps of x and G(s) · L_mel: L1 between log-mel of x and G(s)least-squares GAN (LSGAN) · audio 22,050 Hz · mel n_fft 1024, hop 256, 80 bands
fig.HiFi-GAN V1 on one training segment. An 80 × 32 mel becomes 512 × 32 channels. Upsampling by 8, 8, 2 and 2 (transposed conv kernels 16, 16, 4, 4) reaches 32 × 8192, and a final conv with tanh gives 8192 samples (32 frames × hop 256). Each MRF averages three residual blocks with kernels 3, 7 and 11. The multi-period discriminator folds the waveform into a (T/p) × p grid, so its (5, 1) kernels compare samples p apart. Five period sub-discriminators and three multi-scale ones (raw, 2× and 4× average-pooled audio) score both real and generated audio.

Discriminators

Multi-period discriminator (MPD). It has five sub-discriminators, one per period p∈{2,3,5,7,11}p\in\{2,3,5,7,11\}. Each reflect-pads the waveform to a multiple of pp and reshapes it from length TT to a 2-D map of height T/pT/p and width pp. Its 2-D convolutions have kernels of width 1 along the period axis, (5, 1), with stride (3, 1), so each column is processed independently. A column holds samples xj,xj+p,xj+2p,…x_j, x_{j+p}, x_{j+2p},\dots. The sub-discriminator therefore sees the signal at every phase of period pp. The periods are primes, to avoid overlap between what the sub-discriminators see.

Xi,j(p)=x~ ip+j,0≤j<p,0≤i<⌈T/p⌉,x~=reflectpad⁡(x,  p⌈T/p⌉−T)X^{(p)}_{i,j}=\tilde x_{\,i p+j},\qquad 0\le j<p,\quad 0\le i<\lceil T/p\rceil,\qquad \tilde x=\operatorname{reflectpad}\big(x,\;p\lceil T/p\rceil-T\big)
X(p)X^{(p)}
2-D input of the period-pp sub-discriminator. Moving one step down a column moves pp samples forward in time
x~\tilde x
waveform padded on the right by reflection so its length is a multiple of pp

Multi-scale discriminator (MSD), taken from MelGAN, has three 1-D sub-discriminators. They see the raw waveform and versions average-pooled by 2 and by 4. These catch consecutive patterns and long-term structure, which the MPD, reading isolated sample columns, can miss. The raw-scale one uses spectral normalisation and the others weight normalisation.

sub-discriminator p = 7: T = 132 → reflect-pad 1 → (19, 7) · signal period P = 71-D waveform · phase j = 0: samples 0, p, 2p, … · ringed: one (5, 1) kernel window · reflect padding2-D view (drawn transposed): → T/p axis, the one the (5, 1) kernel slides along · ↓ phase j = 0 … 6colour: positive / negative amplitude · outlined: kernel window = x[0], x[p], …, x[4p]r(lag) = corr(x[t], x[t + lag]) over 1024 samples · MPD periods 2, 3, 5, 7, 11123456789101112131415161718192021222324+1−1r(7) = 0.96 · r(2) -0.29 · r(3) -0.57 · r(5) -0.29 · r(7) 0.96 · r(11) -0.57grid rows are constant when p is a multiple of P; each period checks a different lag
period p
signal
fig.A three-harmonic test signal with period P = 7 samples and noise 0.2, folded for p = 7. The 132 samples get 1 sample of reflect padding and become a 19 × 7 grid, drawn transposed. Every phase row is nearly constant, and r(7) = corr(x[t], x[t+7]) = 0.96. The other MPD periods see r(2) = −0.29, r(3) = −0.57, r(5) = −0.29 and r(11) = −0.57. Turn on phase drift, a stand-in for a generator with an unsteady pitch: r(7) falls to 0.82 and r(14) from 0.96 to 0.73, because longer lags accumulate more drift. Set P = 5.5: now p = 11 sees the structure (r(11) = 0.96), and p = 7 does not (−0.26).

Losses

LD=∑kE[(Dk(x)−1)2+Dk(G(s))2]LG=∑k(E[(Dk(G(s))−1)2]+λfm E∑i1Ni∥Dki(x)−Dki(G(s))∥1)+λmel E∥ϕ(x)−ϕ(G(s))∥1\begin{aligned}\mathcal L_D&=\sum_{k}\mathbb E\big[(D_k(x)-1)^2+D_k(G(s))^2\big]\\ \mathcal L_G&=\sum_{k}\Big(\mathbb E\big[(D_k(G(s))-1)^2\big]+\lambda_{fm}\,\mathbb E\sum_{i}\tfrac{1}{N_i}\big\lVert D_k^{i}(x)-D_k^{i}(G(s))\big\rVert_1\Big)+\lambda_{mel}\,\mathbb E\big\lVert\phi(x)-\phi(G(s))\big\rVert_1\end{aligned}
DkD_k
the 8 sub-discriminators (5 MPD + 3 MSD), trained with the least-squares GAN objective (see Adversarial loss)
Dki, NiD_k^i,\ N_i
feature map of layer ii of sub-discriminator kk, and its number of elements: the feature-matching loss
ϕ\phi
log-mel spectrogram (n_fft 1024, hop 256, 80 bands), so the mel loss is an L1 reconstruction term on what the acoustic model predicted
λfm, λmel\lambda_{fm},\ \lambda_{mel}
2 and 45

The mel loss anchors the generator to the input from the first step and stabilises early training. The adversarial and feature-matching terms then add the fine detail that an L1 loss on spectra would blur, especially phase and harmonic structure.

Versionh_uUpsamplingMRF kernelsParamsMOS
V15128, 8, 2, 23, 7, 1113.92M4.36
V21288, 8, 2, 23, 7, 110.92M4.23
V32568, 8, 43, 5, 7 (lighter blocks)1.46M4.05
The three generator configurations (Kong et al., 2020). MOS on LJSpeech, where ground-truth audio scored 4.45

Setting

Data
LJSpeech: 13,100 clips, about 24 hours, one female speaker, 22,050 Hz
Baselines (MOS)
WaveNet (mixture of logistics) 4.02, WaveGlow 3.81, MelGAN 3.79
Training
random 8192-sample segments; AdamW with β₁ = 0.8, β₂ = 0.99, learning rate 2 × 10⁻⁴ decayed by 0.999 per epoch
Speed
all versions run far faster than real time on a GPU; the small V2 and V3 also run faster than real time on a CPU
Ablation
removing the MPD caused the largest quality drop of any component
p8192 mod preflect pad2-D shape (T/p, p)
2004096 × 2
3212731 × 3
5231639 × 5
7251171 × 7
1183745 × 11
Worked example: MPD inputs for one 8192-sample training segment (computed with numpy)

On the generator side, the same segment is 8192/256=328192/256=32 mel frames, and the stages run 32→256→2048→4096→819232\to256\to2048\to4096\to8192 samples. One second of audio is 22050/256≈8622050/256\approx86 frames.

python
1import numpy as np
2
3def fold(x, p):
4 """MPD input: reflect-pad a 1-D waveform to a multiple of p, view as (T/p, p)."""
5 T = len(x)
6 if T % p:
7 x = np.pad(x, (0, p - T % p), mode="reflect") # same as F.pad(..., "reflect") in PyTorch
8 return x.reshape(-1, p) # row i holds x[i*p : i*p + p]
9
10sr, f0 = 22050, 220.5 # a 100-sample pitch period
11x = np.sin(2 * np.pi * f0 * np.arange(8192) / sr) # one 8192-sample training segment
12for p in [2, 3, 5, 7, 11]:
13 X = fold(x, p)
14 # a (5, 1) kernel sees X[i:i+5, j] = x[t], x[t+p], ..., x[t+4p]
15 r = np.corrcoef(X[:-1].ravel(), X[1:].ravel())[0, 1] # neighbours along the kernel axis
16 print(p, X.shape, f"corr(x[t], x[t+p]) = {r:.3f}")
17# 2 (4096, 2) 0.992 · 3 (2731, 3) 0.982 · 5 (1639, 5) 0.951 · 7 (1171, 7) 0.905 · 11 (745, 11) 0.771
The MPD fold in numpy (runs as is); in PyTorch it is F.pad(x, (0, n_pad), "reflect") followed by x.view(b, c, t // p, p)

Use when

  • +One parallel forward pass: orders of magnitude faster than autoregressive vocoders
  • +V1 came close to ground truth (4.36 vs 4.45 MOS) on LJSpeech
  • +V2 has under 1M parameters, practical on CPU and edge devices

Avoid when

  • −GAN training: 8 sub-discriminators and many steps, sensitive to hyperparameters
  • −Tied to its mel configuration; a new sample rate or hop needs retraining
  • −Quality drops on mismatched predicted mels unless fine-tuned

Check yourself

A 2000-sample waveform enters the period-7 sub-discriminator. What 2-D shape does it get?

Check yourself

What does the multi-scale discriminator see that the multi-period discriminator does not?

Remember

  • ✓Generator: conv → transposed convs ×8, ×8, ×2, ×2 (= hop 256) each followed by MRF (kernels 3, 7, 11; dilations 1, 3, 5) → tanh.
  • ✓MPD: reflect-pad, fold T→(T/p)×pT\to(T/p)\times p for p=2,3,5,7,11p=2,3,5,7,11; (5, 1) kernels compare samples pp apart.
  • ✓MSD (from MelGAN): raw, 2× and 4× average-pooled waveforms.
  • ✓LG\mathcal L_G = LSGAN adversarial + 2 · feature matching + 45 · mel L1.
  • ✓LJSpeech MOS: V1 4.36 (13.92M params), V2 4.23 (0.92M), V3 4.05; ground truth 4.45.
← back to Deep Learning roadmap