# FastSpeech 2

> FastSpeech without the teacher: train on real mels with forced-aligner durations, and add a variance adaptor that predicts duration, pitch and energy, quantises each into 256 bins and adds their embeddings to the frames.

Source: https://www.mlfoundry.in/learn/fastspeech-2  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**FastSpeech 2:** is a non-autoregressive acoustic model that maps phonemes to a mel spectrogram in one parallel pass. It was introduced by Ren, Hu, Tan, Qin, Zhao, Zhao and Liu in "FastSpeech 2: Fast and High-Quality End-to-End Text to Speech" (ICLR 2021). It keeps the **FastSpeech** encoder, length regulator and decoder, but drops knowledge distillation and adds a **variance adaptor** that conditions the decoder on duration, pitch and energy. A variant, **FastSpeech 2s**, generates the waveform directly from text.

**FastSpeech** learned from an autoregressive Transformer TTS teacher. Its training targets were the teacher's generated mels, and its phoneme durations came from the teacher's attention maps. That made training a two-stage pipeline. The distilled mels were also smoother and less detailed than real speech, which capped quality, and attention-derived durations were imprecise.

The deeper problem is that TTS is **one-to-many**. One phoneme sequence can be spoken with many rhythms, melodies and loudness patterns. A model trained with an L1 or L2 loss on mels then averages over them and sounds flat. FastSpeech 2's answer is to give the decoder the missing information explicitly. In training it receives the ground-truth duration, pitch and energy. At inference it receives the predicted values.

## The variance adaptor

> Figure: FastSpeech 2 on 4 phonemes. The encoder gives 4 hidden vectors. The duration predictor's 3, 5, 4 and 8 frames expand them to 20 frames through the length regulator. The pitch predictor outputs a 10-scale wavelet pitch spectrogram that is inverted to F0, quantised into 256 log-scale bins and embedded. Energy (the L2 norm of each STFT frame) is quantised into 256 uniform bins and embedded. Both embeddings are added to the frames, and the mel decoder emits an 80 × 20 mel that the vocoder turns into 20 × 256 = 5120 samples. FastSpeech 2s (dashed) decodes the waveform directly.

The adaptor sits between the encoder and the mel decoder and has three predictors. Each predictor has the same small structure: two 1D convolutions with kernel size 3, each followed by ReLU, layer normalisation and dropout, then a linear layer that outputs one scalar per position. The duration predictor runs on phonemes. The pitch and energy predictors run on frames, after the length regulator.

$$
H_1=\mathrm{LR}(H,d),\qquad H_2=H_1+E_p\big[q_p(p)\big],\qquad H_3=H_2+E_e\big[q_e(e)\big]
$$

- $H,\,d$: encoder output ($N\times256$) and per-phoneme durations in frames. The length regulator repeats row $i$ of $H$ $d_i$ times, giving $T=\sum_i d_i$ frames
- $p,\,e$: frame-level pitch (F0) and energy. Ground truth during training, predictor output at inference
- $q_p,\,q_e$: quantisers into 256 bins: log-spaced for pitch, uniform for energy
- $E_p,\,E_e$: learned embedding tables ($256\times256$) whose rows are added to every frame's hidden state

$$
\mathcal L=\underbrace{\lVert \hat Y-Y\rVert_1}_{\text{mel}}+\underbrace{\lVert \log\hat d-\log d\rVert_2^2}_{\text{duration}}+\underbrace{\lVert \hat p-p\rVert_2^2}_{\text{pitch}}+\underbrace{\lVert \hat e-e\rVert_2^2}_{\text{energy}},\qquad e_t=\big\lVert\,|X_t|\,\big\rVert_2
$$

- $Y,\,\hat Y$: ground-truth and predicted 80-band mel spectrograms; mean absolute error
- $d$: durations from the Montreal Forced Aligner, not from a teacher's attention; predicted in the log domain
- $|X_t|$: STFT magnitude of frame $t$; energy is its L2 norm

**Pitch** is the hardest target because F0 contours vary a lot. The paper extracts F0 with PyWorld, linearly interpolates over unvoiced frames, takes the log and normalises each utterance to zero mean and unit variance. It then decomposes the contour with a **continuous wavelet transform** (CWT) into a 10-scale pitch spectrogram. The predictor outputs that spectrogram plus the utterance's mean and standard deviation, and an inverse CWT turns it back into a contour. Later implementations often skip the CWT and regress F0 directly, sometimes averaged per phoneme as in FastPitch (Łańcucki, 2021).

**Setup in the paper**

- **Data:** LJSpeech: 13,100 clips, about 24 h, one female English speaker, 22,050 Hz
- **Features:** 80-band mel, frame size 1024, hop 256 (86.13 frames per second)
- **Hidden size:** 256 in the encoder, decoder and variance embeddings; predictors use kernel size 3
- **Quantisation:** pitch and energy each into 256 values: pitch on a log scale, energy uniformly
- **Aligner and vocoder:** Montreal Forced Aligner for durations; Parallel WaveGAN turns mels into audio
- **Results:** 3× faster training than FastSpeech; in MOS tests on LJSpeech it beat FastSpeech and matched or exceeded autoregressive Tacotron 2 and Transformer TTS

## Quantised pitch, live

> Figure: An 80-frame F0 contour (0.93 s) with a 7-frame unvoiced gap filled by interpolation. At the defaults (256 log bins over 80–400 Hz, no shift) each bin is 10.88 cents wide, the RMS quantisation error is 3.02 cents (max 5.31), and the 144.2–255.2 Hz contour uses 48 distinct pitch embeddings. With 16 bins the error rises to 43.97 cents RMS (max 83.72) and only 7 embeddings are used. Shifting +12 semitones pushes 34 frames above 400 Hz, where they all clamp to bin 255 (398.7 Hz). Range ×2 widens the contour to 107.1–335.3 Hz (62 bins), and range ×0 gives a monotone that uses a single bin.

Because pitch enters the decoder as an embedding lookup, prosody becomes a control input. Multiply the predicted contour by $2^{k/12}$ to shift it $k$ semitones. Stretch its log range around the mean to make the speech more or less expressive. Scale the predicted durations to change speed. The decoder has seen many pitch values in training, so it renders them without retraining. It cannot render values outside the quantiser's range, though.

**Worked example: quantising one frame (256 log bins over 80–400 Hz)**

1. Log range: $\ln(400/80)=1.6094$. Divided into 256 bins, each is $0.006287$ in $\ln f$, i.e. $2786.3/256=10.88$ cents.
2. Frame at $f=220$ Hz: $256\cdot\ln(220/80)/1.6094=160.91$, so bin $160$. Its centre is $80\,e^{160.5\times0.006287}=219.44$ Hz, an error of $-4.43$ cents. Frame $t$ gets $E_p[160]$.
3. Shift $+2$ semitones: $220\cdot2^{2/12}=246.94$ Hz → bin $179$ (centre $247.28$ Hz, $+2.37$ cents). One semitone is $100/10.88=9.19$ bins.
4. Timing: 80 frames $\times\,256/22{,}050=0.929$ s of audio. The decoder emits all 80 frames at once.

```python
import numpy as np

sr, n_fft, hop, n_bins = 22050, 1024, 256, 256
t = np.arange(sr) / sr                                   # 1 s test signal
f0 = 180 + 40 * np.sin(2 * np.pi * 1.5 * t)              # gliding pitch, 140-220 Hz
y = 0.5 * np.sin(2 * np.pi * np.cumsum(f0) / sr) * np.hanning(sr)

# Energy target: L2 norm of each STFT magnitude frame (FastSpeech 2's definition)
frames = np.lib.stride_tricks.sliding_window_view(np.pad(y, n_fft // 2), n_fft)[::hop]
mag = np.abs(np.fft.rfft(frames * np.hanning(n_fft), axis=1))
energy = np.linalg.norm(mag, axis=1)                     # (T,)
f0_frames = f0[np.minimum(np.arange(len(frames)) * hop, sr - 1)]   # stand-in for a pitch tracker

# Quantise: 256 log-spaced pitch bins, 256 uniform energy bins (equal-width cells)
p_edges = np.exp(np.linspace(np.log(80), np.log(400), n_bins + 1))[1:-1]   # 255 inner edges
e_edges = np.linspace(energy.min(), energy.max(), n_bins + 1)[1:-1]
p_idx = np.digitize(f0_frames, p_edges)                  # ints in [0, 255] -> nn.Embedding(256, 256)
e_idx = np.digitize(energy, e_edges)
print(len(frames), p_idx.min(), p_idx.max(), e_idx.min(), e_idx.max())   # 87 89 160 0 255

# Pitch control at inference: shift the predicted contour by +2 semitones, then re-quantise
print(np.digitize(220.0, p_edges), np.digitize(220.0 * 2 ** (2 / 12), p_edges))   # 160 179
```

FastSpeech 2 variance targets in numpy: frame energy from an STFT, then 256-bin pitch and energy quantisation (run here; it prints 87 frames, pitch bins 89–160, and 160 → 179 for a +2 semitone shift)

## FastSpeech 2s

FastSpeech 2s removes the mel stage at inference. A waveform decoder upsamples the adapted frames with transposed convolutions and uses WaveNet-style gated, non-causal convolutions. It is trained adversarially with a Parallel WaveGAN-style discriminator. Full utterances do not fit in memory, so it trains on short random audio slices. The mel decoder is kept during training as an auxiliary task that helps learn text features. 2s is faster at inference than FastSpeech 2 plus a vocoder, but the paper rated its quality slightly lower. Most later systems kept the two-stage design with a separate **HiFi-GAN** vocoder.

> **The bins are fixed at training time:** The quantiser's range comes from training-set statistics. A large upward shift or a new speaker with a higher voice puts frames outside it, and they all collapse into the top bin (see the +12 semitone setting in the figure). Clamp control values to the training range, or normalise pitch per speaker before quantising.

> **Why this fixes the one-to-many problem:** With the true duration, pitch and energy as input, the remaining mapping from phonemes to mel is close to deterministic, so an L1 loss no longer averages incompatible targets. The variability is moved into three small predictors. Those predictors still use MSE, so their outputs tend toward the average prosody, which is why stochastic duration and pitch models came next (**Neural text-to-speech**).

**Strengths**

- No teacher: trains on real mels and is 3× faster to train than FastSpeech
- Parallel and robust: explicit durations mean no skipped or repeated words
- Pitch, energy and speed are directly controllable at inference

**Limitations**

- Needs an external forced aligner and pitch extractor; their errors become training targets
- MSE-trained predictors give average, somewhat flat prosody
- Still two stages: a separate vocoder unless the harder-to-train 2s variant is used

- **Low-latency production TTS:** One parallel pass per sentence plus a fast GAN vocoder keeps latency low enough for assistants and screen readers.
- **Prosody editing:** Speed, pitch shift and pitch range are edited by changing the predicted durations and contours before the decoder runs.
- **Multi-speaker and expressive TTS:** A speaker or emotion embedding is added to the encoder output, and the variance predictors learn that speaker's or emotion's prosody. FastSpeech 2 is also a common acoustic model in toolkits such as ESPnet.

**Question:** FastSpeech 2 uses ground-truth pitch and energy in training but predicted values at inference. What does this buy?

- A. It removes the need for a vocoder
- B. The decoder learns a nearly deterministic mapping given prosody, so it is not forced to average many possible outputs
- C. It makes the model autoregressive
- D. It lets the model skip the length regulator

**Answer:** B. The decoder learns a nearly deterministic mapping given prosody, so it is not forced to average many possible outputs — Variance information resolves most of the one-to-many ambiguity. The decoder learns "render these phonemes with this pitch and energy", and the predictors learn the prosody separately. This is the same teacher-forcing idea used for durations in **FastSpeech**.

**Question:** Pitch is quantised into 256 log-spaced bins over 80–400 Hz. What is the largest quantisation error for an in-range frame?

- A. about 0.4 cents
- B. about 5.4 cents
- C. about 10.9 cents
- D. about 1 semitone

**Answer:** B. about 5.4 cents — The range is $1200\log_2 5=2786.3$ cents, so each bin is $10.88$ cents wide. Using the bin centre, the error is at most half a bin, $5.44$ cents. The figure's contour reaches 5.31.

**Key points**

- FastSpeech 2 (Ren et al., ICLR 2021) trains on real mels with Montreal Forced Aligner durations; no teacher, 3× faster training.
- Variance adaptor: duration → length regulator → $+E_p[q_p(p)]$ → $+E_e[q_e(e)]$; ground truth in training, predictions at inference.
- Each predictor: 2 × conv1D (kernel 3) + ReLU + layer norm + dropout + linear.
- Pitch: interpolated, log, per-utterance normalised F0 modelled as a 10-scale CWT spectrogram; 256 log bins. Energy: L2 norm of STFT frames, 256 uniform bins.
- Prosody becomes a control input (shift, range, speed); FastSpeech 2s decodes the waveform directly with an adversarial loss.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
