Definition
FastSpeech 2 is a non-autoregressive acoustic model that maps phonemes to a mel spectrogram in one parallel pass. It was introduced by Ren, Hu, Tan, Qin, Zhao, Zhao and Liu in "FastSpeech 2: Fast and High-Quality End-to-End Text to Speech" (ICLR 2021). It keeps the FastSpeech encoder, length regulator and decoder, but drops knowledge distillation and adds a variance adaptor that conditions the decoder on duration, pitch and energy. A variant, FastSpeech 2s, generates the waveform directly from text.
FastSpeech learned from an autoregressive Transformer TTS teacher. Its training targets were the teacher's generated mels, and its phoneme durations came from the teacher's attention maps. That made training a two-stage pipeline. The distilled mels were also smoother and less detailed than real speech, which capped quality, and attention-derived durations were imprecise.
The deeper problem is that TTS is one-to-many. One phoneme sequence can be spoken with many rhythms, melodies and loudness patterns. A model trained with an L1 or L2 loss on mels then averages over them and sounds flat. FastSpeech 2's answer is to give the decoder the missing information explicitly. In training it receives the ground-truth duration, pitch and energy. At inference it receives the predicted values.
The variance adaptor
The adaptor sits between the encoder and the mel decoder and has three predictors. Each predictor has the same small structure: two 1D convolutions with kernel size 3, each followed by ReLU, layer normalisation and dropout, then a linear layer that outputs one scalar per position. The duration predictor runs on phonemes. The pitch and energy predictors run on frames, after the length regulator.
- encoder output () and per-phoneme durations in frames. The length regulator repeats row of times, giving frames
- frame-level pitch (F0) and energy. Ground truth during training, predictor output at inference
- quantisers into 256 bins: log-spaced for pitch, uniform for energy
- learned embedding tables () whose rows are added to every frame's hidden state
- ground-truth and predicted 80-band mel spectrograms; mean absolute error
- durations from the Montreal Forced Aligner, not from a teacher's attention; predicted in the log domain
- STFT magnitude of frame ; energy is its L2 norm
Pitch is the hardest target because F0 contours vary a lot. The paper extracts F0 with PyWorld, linearly interpolates over unvoiced frames, takes the log and normalises each utterance to zero mean and unit variance. It then decomposes the contour with a continuous wavelet transform (CWT) into a 10-scale pitch spectrogram. The predictor outputs that spectrogram plus the utterance's mean and standard deviation, and an inverse CWT turns it back into a contour. Later implementations often skip the CWT and regress F0 directly, sometimes averaged per phoneme as in FastPitch (Łańcucki, 2021).
Setup in the paper
- Data
- LJSpeech: 13,100 clips, about 24 h, one female English speaker, 22,050 Hz
- Features
- 80-band mel, frame size 1024, hop 256 (86.13 frames per second)
- Hidden size
- 256 in the encoder, decoder and variance embeddings; predictors use kernel size 3
- Quantisation
- pitch and energy each into 256 values: pitch on a log scale, energy uniformly
- Aligner and vocoder
- Montreal Forced Aligner for durations; Parallel WaveGAN turns mels into audio
- Results
- 3× faster training than FastSpeech; in MOS tests on LJSpeech it beat FastSpeech and matched or exceeded autoregressive Tacotron 2 and Transformer TTS
Quantised pitch, live
Because pitch enters the decoder as an embedding lookup, prosody becomes a control input. Multiply the predicted contour by to shift it semitones. Stretch its log range around the mean to make the speech more or less expressive. Scale the predicted durations to change speed. The decoder has seen many pitch values in training, so it renders them without retraining. It cannot render values outside the quantiser's range, though.
Worked example: quantising one frame (256 log bins over 80–400 Hz)
- 1Log range: . Divided into 256 bins, each is in , i.e. cents.
- 2Frame at Hz: , so bin . Its centre is Hz, an error of cents. Frame gets .
- 3Shift semitones: Hz → bin (centre Hz, cents). One semitone is bins.
- 4Timing: 80 frames s of audio. The decoder emits all 80 frames at once.
1import numpy as np23sr, n_fft, hop, n_bins = 22050, 1024, 256, 2564t = np.arange(sr) / sr # 1 s test signal5f0 = 180 + 40 * np.sin(2 * np.pi * 1.5 * t) # gliding pitch, 140-220 Hz6y = 0.5 * np.sin(2 * np.pi * np.cumsum(f0) / sr) * np.hanning(sr)78# Energy target: L2 norm of each STFT magnitude frame (FastSpeech 2's definition)9frames = np.lib.stride_tricks.sliding_window_view(np.pad(y, n_fft // 2), n_fft)[::hop]10mag = np.abs(np.fft.rfft(frames * np.hanning(n_fft), axis=1))11energy = np.linalg.norm(mag, axis=1) # (T,)12f0_frames = f0[np.minimum(np.arange(len(frames)) * hop, sr - 1)] # stand-in for a pitch tracker1314# Quantise: 256 log-spaced pitch bins, 256 uniform energy bins (equal-width cells)15p_edges = np.exp(np.linspace(np.log(80), np.log(400), n_bins + 1))[1:-1] # 255 inner edges16e_edges = np.linspace(energy.min(), energy.max(), n_bins + 1)[1:-1]17p_idx = np.digitize(f0_frames, p_edges) # ints in [0, 255] -> nn.Embedding(256, 256)18e_idx = np.digitize(energy, e_edges)19print(len(frames), p_idx.min(), p_idx.max(), e_idx.min(), e_idx.max()) # 87 89 160 0 2552021# Pitch control at inference: shift the predicted contour by +2 semitones, then re-quantise22print(np.digitize(220.0, p_edges), np.digitize(220.0 * 2 ** (2 / 12), p_edges)) # 160 179FastSpeech 2s
FastSpeech 2s removes the mel stage at inference. A waveform decoder upsamples the adapted frames with transposed convolutions and uses WaveNet-style gated, non-causal convolutions. It is trained adversarially with a Parallel WaveGAN-style discriminator. Full utterances do not fit in memory, so it trains on short random audio slices. The mel decoder is kept during training as an auxiliary task that helps learn text features. 2s is faster at inference than FastSpeech 2 plus a vocoder, but the paper rated its quality slightly lower. Most later systems kept the two-stage design with a separate HiFi-GAN vocoder.
Use when
- +No teacher: trains on real mels and is 3× faster to train than FastSpeech
- +Parallel and robust: explicit durations mean no skipped or repeated words
- +Pitch, energy and speed are directly controllable at inference
Avoid when
- −Needs an external forced aligner and pitch extractor; their errors become training targets
- −MSE-trained predictors give average, somewhat flat prosody
- −Still two stages: a separate vocoder unless the harder-to-train 2s variant is used
Where it's used
Low-latency production TTS
One parallel pass per sentence plus a fast GAN vocoder keeps latency low enough for assistants and screen readers.
Prosody editing
Speed, pitch shift and pitch range are edited by changing the predicted durations and contours before the decoder runs.
Multi-speaker and expressive TTS
A speaker or emotion embedding is added to the encoder output, and the variance predictors learn that speaker's or emotion's prosody. FastSpeech 2 is also a common acoustic model in toolkits such as ESPnet.
Check yourself
FastSpeech 2 uses ground-truth pitch and energy in training but predicted values at inference. What does this buy?
Check yourself
Pitch is quantised into 256 log-spaced bins over 80–400 Hz. What is the largest quantisation error for an in-range frame?
Remember
- ✓FastSpeech 2 (Ren et al., ICLR 2021) trains on real mels with Montreal Forced Aligner durations; no teacher, 3× faster training.
- ✓Variance adaptor: duration → length regulator → → ; ground truth in training, predictions at inference.
- ✓Each predictor: 2 × conv1D (kernel 3) + ReLU + layer norm + dropout + linear.
- ✓Pitch: interpolated, log, per-utterance normalised F0 modelled as a 10-scale CWT spectrogram; 256 log bins. Energy: L2 norm of STFT frames, 256 uniform bins.
- ✓Prosody becomes a control input (shift, range, speed); FastSpeech 2s decodes the waveform directly with an adversarial loss.