# HiFi-GAN

> HiFi-GAN is a GAN vocoder: transposed convolutions with multi-receptive-field fusion upsample a mel spectrogram 256×, judged by period discriminators (p = 2, 3, 5, 7, 11) and multi-scale discriminators, with a mel L1 loss weighted 45.

Source: https://www.mlfoundry.in/learn/hifi-gan  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**HiFi-GAN:** is a non-autoregressive neural vocoder trained as a **Generative adversarial network (GAN)**. A fully convolutional generator maps a mel spectrogram to a waveform in one forward pass. Two families of discriminators, multi-period and multi-scale, judge the result. Introduced by Jungil Kong, Jaehyeon Kim and Jaekyoung Bae (Kakao Enterprise), "HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis" (NeurIPS 2020).

A vocoder turns the acoustic model's mel spectrogram into audio (see **Text-to-speech** for the pipeline). At 22,050 Hz with a 256-sample hop, every mel frame must become 256 samples. The mel throws away phase, so the vocoder has to invent a phase that sounds natural. Errors in periodic structure (pitch harmonics) are especially audible as buzz or metallic artefacts.

Autoregressive vocoders such as **WaveNet** and **WaveRNN** sound good but generate one sample at a time. Flow-based WaveGlow is parallel but large. Earlier GAN vocoders such as MelGAN (Kumar et al., 2019) were fast, but their quality lagged behind. HiFi-GAN's argument is that the discriminator must look explicitly at **periodic** patterns, because speech is a sum of sinusoids with different periods.

## Generator: upsampling with multi-receptive-field fusion

The generator starts with a 7-tap convolution to $h_u$ channels (512 in V1). Four transposed-convolution stages (see **Transposed convolution**) then upsample by 8, 8, 2 and 2, a total of 256, the hop length. Each stage halves the channels. After each upsampling comes a **multi-receptive-field fusion (MRF)** module: residual blocks with kernel sizes 3, 7 and 11 and dilations 1, 3 and 5 (see **Dilated convolution**) run in parallel, and their outputs are averaged. Short kernels capture fine detail and long dilated ones capture longer patterns, all at the same resolution. A final convolution and tanh give the waveform.

> Figure: HiFi-GAN V1 on one training segment. An 80 × 32 mel becomes 512 × 32 channels. Upsampling by 8, 8, 2 and 2 (transposed conv kernels 16, 16, 4, 4) reaches 32 × 8192, and a final conv with tanh gives 8192 samples (32 frames × hop 256). Each MRF averages three residual blocks with kernels 3, 7 and 11. The multi-period discriminator folds the waveform into a (T/p) × p grid, so its (5, 1) kernels compare samples p apart. Five period sub-discriminators and three multi-scale ones (raw, 2× and 4× average-pooled audio) score both real and generated audio.

## Discriminators

**Multi-period discriminator (MPD).** It has five sub-discriminators, one per period $p\in\{2,3,5,7,11\}$. Each reflect-pads the waveform to a multiple of $p$ and reshapes it from length $T$ to a 2-D map of height $T/p$ and width $p$. Its 2-D convolutions have kernels of width 1 along the period axis, (5, 1), with stride (3, 1), so each column is processed independently. A column holds samples $x_j, x_{j+p}, x_{j+2p},\dots$. The sub-discriminator therefore sees the signal at every phase of period $p$. The periods are primes, to avoid overlap between what the sub-discriminators see.

$$
X^{(p)}_{i,j}=\tilde x_{\,i p+j},\qquad 0\le j<p,\quad 0\le i<\lceil T/p\rceil,\qquad \tilde x=\operatorname{reflectpad}\big(x,\;p\lceil T/p\rceil-T\big)
$$

- $X^{(p)}$: 2-D input of the period-$p$ sub-discriminator. Moving one step down a column moves $p$ samples forward in time
- $\tilde x$: waveform padded on the right by reflection so its length is a multiple of $p$

**Multi-scale discriminator (MSD)**, taken from MelGAN, has three 1-D sub-discriminators. They see the raw waveform and versions average-pooled by 2 and by 4. These catch consecutive patterns and long-term structure, which the MPD, reading isolated sample columns, can miss. The raw-scale one uses spectral normalisation and the others weight normalisation.

> Figure: A three-harmonic test signal with period P = 7 samples and noise 0.2, folded for p = 7. The 132 samples get 1 sample of reflect padding and become a 19 × 7 grid, drawn transposed. Every phase row is nearly constant, and r(7) = corr(x[t], x[t+7]) = 0.96. The other MPD periods see r(2) = −0.29, r(3) = −0.57, r(5) = −0.29 and r(11) = −0.57. Turn on phase drift, a stand-in for a generator with an unsteady pitch: r(7) falls to 0.82 and r(14) from 0.96 to 0.73, because longer lags accumulate more drift. Set P = 5.5: now p = 11 sees the structure (r(11) = 0.96), and p = 7 does not (−0.26).

## Losses

$$
\begin{aligned}\mathcal L_D&=\sum_{k}\mathbb E\big[(D_k(x)-1)^2+D_k(G(s))^2\big]\\ \mathcal L_G&=\sum_{k}\Big(\mathbb E\big[(D_k(G(s))-1)^2\big]+\lambda_{fm}\,\mathbb E\sum_{i}\tfrac{1}{N_i}\big\lVert D_k^{i}(x)-D_k^{i}(G(s))\big\rVert_1\Big)+\lambda_{mel}\,\mathbb E\big\lVert\phi(x)-\phi(G(s))\big\rVert_1\end{aligned}
$$

- $D_k$: the 8 sub-discriminators (5 MPD + 3 MSD), trained with the least-squares GAN objective (see **Adversarial loss**)
- $D_k^i,\ N_i$: feature map of layer $i$ of sub-discriminator $k$, and its number of elements: the **feature-matching** loss
- $\phi$: log-mel spectrogram (n_fft 1024, hop 256, 80 bands), so the **mel loss** is an L1 reconstruction term on what the acoustic model predicted
- $\lambda_{fm},\ \lambda_{mel}$: 2 and 45

The mel loss anchors the generator to the input from the first step and stabilises early training. The adversarial and feature-matching terms then add the fine detail that an L1 loss on spectra would blur, especially phase and harmonic structure.

The three generator configurations (Kong et al., 2020). MOS on LJSpeech, where ground-truth audio scored 4.45

| Version | h_u | Upsampling | MRF kernels | Params | MOS |
| --- | --- | --- | --- | --- | --- |
| V1 | 512 | 8, 8, 2, 2 | 3, 7, 11 | 13.92M | 4.36 |
| V2 | 128 | 8, 8, 2, 2 | 3, 7, 11 | 0.92M | 4.23 |
| V3 | 256 | 8, 8, 4 | 3, 5, 7 (lighter blocks) | 1.46M | 4.05 |

**Setting**

- **Data:** LJSpeech: 13,100 clips, about 24 hours, one female speaker, 22,050 Hz
- **Baselines (MOS):** WaveNet (mixture of logistics) 4.02, WaveGlow 3.81, MelGAN 3.79
- **Training:** random 8192-sample segments; AdamW with β₁ = 0.8, β₂ = 0.99, learning rate 2 × 10⁻⁴ decayed by 0.999 per epoch
- **Speed:** all versions run far faster than real time on a GPU; the small V2 and V3 also run faster than real time on a CPU
- **Ablation:** removing the MPD caused the largest quality drop of any component

Worked example: MPD inputs for one 8192-sample training segment (computed with numpy)

| p | 8192 mod p | reflect pad | 2-D shape (T/p, p) |
| --- | --- | --- | --- |
| 2 | 0 | 0 | 4096 × 2 |
| 3 | 2 | 1 | 2731 × 3 |
| 5 | 2 | 3 | 1639 × 5 |
| 7 | 2 | 5 | 1171 × 7 |
| 11 | 8 | 3 | 745 × 11 |

On the generator side, the same segment is $8192/256=32$ mel frames, and the stages run $32\to256\to2048\to4096\to8192$ samples. One second of audio is $22050/256\approx86$ frames.

```python
import numpy as np

def fold(x, p):
    """MPD input: reflect-pad a 1-D waveform to a multiple of p, view as (T/p, p)."""
    T = len(x)
    if T % p:
        x = np.pad(x, (0, p - T % p), mode="reflect")   # same as F.pad(..., "reflect") in PyTorch
    return x.reshape(-1, p)                              # row i holds x[i*p : i*p + p]

sr, f0 = 22050, 220.5                                    # a 100-sample pitch period
x = np.sin(2 * np.pi * f0 * np.arange(8192) / sr)       # one 8192-sample training segment
for p in [2, 3, 5, 7, 11]:
    X = fold(x, p)
    # a (5, 1) kernel sees X[i:i+5, j] = x[t], x[t+p], ..., x[t+4p]
    r = np.corrcoef(X[:-1].ravel(), X[1:].ravel())[0, 1]   # neighbours along the kernel axis
    print(p, X.shape, f"corr(x[t], x[t+p]) = {r:.3f}")
# 2 (4096, 2) 0.992 · 3 (2731, 3) 0.982 · 5 (1639, 5) 0.951 · 7 (1171, 7) 0.905 · 11 (745, 11) 0.771
```

The MPD fold in numpy (runs as is); in PyTorch it is F.pad(x, (0, n_pad), "reflect") followed by x.view(b, c, t // p, p)

> **Why fold instead of downsample:** Average pooling (the MSD) mixes neighbouring samples and acts as a low-pass filter. Folding by $p$ keeps every sample but regroups them, so a column is the signal sampled at stride $p$ with a fixed phase. Gradients reach every sample of the waveform, and each sub-discriminator compares samples at its own distance $p$. Using five different primes covers many such distances with little overlap.

> **Train on the mels you will actually get:** A vocoder trained only on ground-truth mels can buzz when fed the smoother mels predicted by an acoustic model. The paper fine-tuned HiFi-GAN on mel spectrograms predicted by **Tacotron 2** (with teacher forcing, so they stay time-aligned with the real audio) and saw the end-to-end quality improve. Mel parameters (sample rate, n_fft, hop, mel range) must also match exactly between the two models.

**Strengths**

- One parallel forward pass: orders of magnitude faster than autoregressive vocoders
- V1 came close to ground truth (4.36 vs 4.45 MOS) on LJSpeech
- V2 has under 1M parameters, practical on CPU and edge devices

**Limitations**

- GAN training: 8 sub-discriminators and many steps, sensitive to hyperparameters
- Tied to its mel configuration; a new sample rate or hop needs retraining
- Quality drops on mismatched predicted mels unless fine-tuned

**Question:** A 2000-sample waveform enters the period-7 sub-discriminator. What 2-D shape does it get?

- A. 285 × 7 after cropping 5 samples
- B. 286 × 7 after reflect-padding 2 samples
- C. 7 × 2000
- D. 2000 × 7

**Answer:** B. 286 × 7 after reflect-padding 2 samples — $2000 \bmod 7=5$, so $7-5=2$ samples of reflect padding give 2002 samples, and $2002/7=286$ rows of width 7.

**Question:** What does the multi-scale discriminator see that the multi-period discriminator does not?

- A. The mel spectrogram input
- B. Consecutive samples and smoothed, downsampled versions of the waveform, which expose continuous and long-term structure
- C. Only the phase of each sample
- D. Text features from the acoustic model

**Answer:** B. Consecutive samples and smoothed, downsampled versions of the waveform, which expose continuous and long-term structure — The MPD's (5, 1) kernels only combine samples p apart within a column. The MSD runs 1-D convolutions over consecutive samples at 1×, 2× and 4× average-pooled scales.

**Key points**

- Generator: conv → transposed convs ×8, ×8, ×2, ×2 (= hop 256) each followed by MRF (kernels 3, 7, 11; dilations 1, 3, 5) → tanh.
- MPD: reflect-pad, fold $T\to(T/p)\times p$ for $p=2,3,5,7,11$; (5, 1) kernels compare samples $p$ apart.
- MSD (from MelGAN): raw, 2× and 4× average-pooled waveforms.
- $\mathcal L_G$ = LSGAN adversarial + 2 · feature matching + 45 · mel L1.
- LJSpeech MOS: V1 4.36 (13.92M params), V2 4.23 (0.92M), V3 4.05; ground truth 4.45.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
