Definition
FastSpeech is a non-autoregressive text-to-speech acoustic model. It predicts how many mel frames each phoneme lasts, expands the phoneme sequence to that length, and generates every mel frame in parallel. It was introduced by Ren, Ruan, Tan, Qin, Zhao, Zhao and Liu in "FastSpeech: Fast, Robust and Controllable Text to Speech" (NeurIPS 2019). The phoneme durations are extracted from the attention of an autoregressive teacher model.
Autoregressive models such as Tacotron 2 generate mel frames one at a time, each conditioned on the last. A 5-second utterance at a 12.5 ms hop takes 400 sequential decoder steps, so latency grows with length. Their soft attention can also skip or repeat words, and they give no direct handle on speaking rate.
FastSpeech removes the attention between text and speech. It makes the alignment explicit: phoneme gets exactly frames, in order. Once the frame count is known, a feed-forward network can produce all frames at once. The length regulator itself is introduced in Text-to-speech. This article covers where the durations come from and what parallel generation buys.
Architecture
The model is built from feed-forward Transformer (FFT) blocks. Each block is multi-head Self-attention followed by a two-layer 1-D convolution (kernel 3) in place of the position-wise feed-forward layer of a Transformer encoder, with residual connections and layer normalisation. The convolution suits speech, where neighbouring frames and phonemes are closely related. Six FFT blocks run on the phoneme side and six on the mel side, with hidden size 384 and 2 attention heads.
- phoneme states from the encoder FFT blocks
- durations in mel frames: teacher-extracted during training, predicted at inference
- speed control: is 1.5× slower, twice as fast
- duration loss, MSE in the log domain; implementations add an offset such as so that is allowed
Durations from a teacher's attention
There are no human labels for phoneme durations, so FastSpeech first trains an autoregressive Transformer TTS model (Li et al., 2019) as a teacher. Its encoder–decoder attention heads learn soft alignments between mel frames and phonemes. Not all heads are useful: some are diagonal, others are blurry or look at punctuation. FastSpeech chooses the head whose alignment is most diagonal, measured by the focus rate.
- teacher attention of mel frame on phoneme ; each row sums to 1
- numbers of mel frames and phonemes
- focus rate: 1 for a hard one-hot alignment, for uniform attention
- duration of phoneme : the number of frames whose strongest attention is on it.
The teacher is used a second time for sequence-level Knowledge distillation. FastSpeech is trained on the mel spectrograms the teacher generates for each text, not on the recordings. The teacher's outputs are more consistent from one example to the next, which makes the one-to-many mapping from text to speech easier to fit for a model that predicts every frame independently.
Setup and results (Ren et al., 2019)
- Data
- LJSpeech: 13,100 clips, about 24 h, one female English speaker, 22,050 Hz; text converted to phonemes
- Model
- 6 + 6 FFT blocks, hidden size 384, 2 attention heads, convolution kernel 3 with 1536 inner channels
- Teacher
- autoregressive Transformer TTS: gives the attention for durations and the distilled mel targets
- Speed
- about 270× faster mel generation and 38× faster end-to-end synthesis (with WaveGlow) than the Transformer TTS baseline
- Robustness
- no repeated or skipped words on the paper's set of 50 particularly hard sentences, where the autoregressive models made errors
- Control
- α from 0.5 to 1.5 changed speed smoothly; lengthening the space tokens added pauses between words
Extracting durations, live
Worked example
| Frame | S | IY1 | T | max | argmax |
|---|---|---|---|---|---|
| 1 | 0.80 | 0.15 | 0.05 | 0.80 | S |
| 2 | 0.60 | 0.30 | 0.10 | 0.60 | S |
| 3 | 0.25 | 0.65 | 0.10 | 0.65 | IY1 |
| 4 | 0.10 | 0.80 | 0.10 | 0.80 | IY1 |
| 5 | 0.10 | 0.55 | 0.35 | 0.55 | IY1 |
| 6 | 0.05 | 0.30 | 0.65 | 0.65 | T |
| 7 | 0.05 | 0.10 | 0.85 | 0.85 | T |
- 1Focus rate: . A flatter head, , has the same argmaxes but , so this head is chosen.
- 2Durations from argmax counts: , summing to .
- 3Length regulator at : , rounded half up to , giving 11 frames. Banker's rounding, as in
torch.roundornp.round, would give 4 for IY1 and 10 frames in total. - 4At : , 4 frames. At hop 256 and 22,050 Hz, 7 frames are 81.3 ms of audio.
1import numpy as np23def focus_rate(a): # a: (S frames, T phonemes), each row sums to 14 return a.max(axis=1).mean()56def extract_durations(heads): # heads: (H, S, T) teacher encoder-decoder attention7 best = int(np.argmax([focus_rate(a) for a in heads]))8 d = np.bincount(heads[best].argmax(axis=1), minlength=heads.shape[2])9 return best, d # d sums to S: every frame is given to one phoneme1011def length_regulator(h, d, alpha=1.0): # h: (T, dim) phoneme states from the encoder12 reps = np.floor(alpha * d + 0.5).astype(int) # round half up13 return np.repeat(h, reps, axis=0) # (sum(reps), dim)1415# teacher head for "seat" = S IY1 T, 7 mel frames16sharp = np.array([[.80, .15, .05], [.60, .30, .10], [.25, .65, .10], [.10, .80, .10],17 [.10, .55, .35], [.05, .30, .65], [.05, .10, .85]])18blurry = 0.5 * sharp + 0.5 / 3 # same argmaxes, flatter rows19best, d = extract_durations(np.stack([blurry, sharp]))20print(best, d, round(focus_rate(sharp), 4), round(focus_rate(blurry), 4)) # 1 [2 3 2] 0.7 0.51672122h = np.random.default_rng(0).normal(size=(3, 384))23for alpha in (0.5, 1.0, 1.5):24 print(alpha, length_regulator(h, d, alpha).shape) # (4, 384) (7, 384) (11, 384)Use when
- +Parallel mel generation: latency barely grows with utterance length on a GPU
- +No skipped or repeated words, since alignment is explicit and monotonic
- +Speed and pauses are controllable through α and the durations of space tokens
Avoid when
- −Needs a trained autoregressive teacher, which makes the training pipeline longer
- −Duration and mel quality are bounded by the teacher, and distillation loses some variability
- −Durations are deterministic, so prosody is averaged; pitch and energy are not modelled explicitly
Check yourself
A teacher head assigns all 120 frames of an utterance either to the first or to the last phoneme, each with weight 0.9. What is its focus rate, and is it a good head for durations?
Check yourself
FastSpeech predicts durations summing to 200 frames (hop 256, 22,050 Hz). How many sequential decoder steps does the mel decoder need, and how long is the audio at α = 1?
Remember
- ✓Feed-forward Transformer: 6 FFT blocks on phonemes, a length regulator, 6 FFT blocks on frames; self-attention plus 1-D convolution, hidden size 384.
- ✓Durations come from the teacher head with the highest focus rate ; is the number of frames whose argmax is .
- ✓The duration predictor (two convolutions and a linear layer) is trained with MSE on log durations and used at inference; .
- ✓Sequence-level distillation: train on the teacher's generated mels. About 270× faster mel generation and 38× faster end-to-end synthesis.
- ✓Explicit monotonic alignment: no skipping or repeating; α gives speed control. FastSpeech 2 removes the teacher.