FastSpeech

Ren et al. (2019): a non-autoregressive feed-forward Transformer copies each phoneme state for its predicted duration (length regulator, speed α) and emits all mel frames at once. Durations come from a teacher's attention; 270× faster mel generation.

8 min read680 wordscore0 viewsupdated 2026-10-15

Definition

FastSpeech is a non-autoregressive text-to-speech acoustic model. It predicts how many mel frames each phoneme lasts, expands the phoneme sequence to that length, and generates every mel frame in parallel. It was introduced by Ren, Ruan, Tan, Qin, Zhao, Zhao and Liu in "FastSpeech: Fast, Robust and Controllable Text to Speech" (NeurIPS 2019). The phoneme durations are extracted from the attention of an autoregressive teacher model.

Autoregressive models such as Tacotron 2 generate mel frames one at a time, each conditioned on the last. A 5-second utterance at a 12.5 ms hop takes 400 sequential decoder steps, so latency grows with length. Their soft attention can also skip or repeat words, and they give no direct handle on speaking rate.

FastSpeech removes the attention between text and speech. It makes the alignment explicit: phoneme ii gets exactly did_i frames, in order. Once the frame count is known, a feed-forward network can produce all frames at once. The length regulator itself is introduced in Text-to-speech. This article covers where the durations come from and what parallel generation buys.

Architecture

The model is built from feed-forward Transformer (FFT) blocks. Each block is multi-head Self-attention followed by a two-layer 1-D convolution (kernel 3) in place of the position-wise feed-forward layer of a Transformer encoder, with residual connections and layer normalisation. The convolution suits speech, where neighbouring frames and phonemes are closely related. Six FFT blocks run on the phoneme side and six on the mel side, with hidden size 384 and 2 attention heads.

FastSpeech (Ren et al., 2019) · “fast speech” → 8 phonemes → 69 mel frames, all predicted in one passphonemesF AE1 S TS P IY1 CHembedding384-d+ positions6 × FFT blockself-attention + Conv1Dphoneme sideH: 8 × 384one state per phonemeduration predictor2 × Conv1D (k = 3)+ LN, dropout, linearMSE on log durationdurations d (frames)8109486159Σ d = 69training only: teacherautoregressive TransformerTTS; head with max focusrate → argmax countslength regulatorrepeat h_i ⌊α·d_i⌉ ×α = 1 → 69 framesα = 1.5 → 105 (slower)FAE1STSPIY1CH69 × 384 · one row per mel frame (hop 256 at 22.05 kHz = 11.6 ms)6 × FFT blockmel side, + positionsno causal masklinear384 → 80per framevocoderWaveGlow in the papermel 80 × 69 → audioFFT block = self-attention (2 heads, d = 384) + 2 × Conv1D (kernel 3, 1536 hidden), each with residual + LNvs the autoregressive teacher: 270× faster mel generation, 38× faster end to end (paper’s measurements)
fig.FastSpeech on "fast speech". The 8 phonemes pass through 6 FFT blocks. A duration predictor (two 1-D convolutions and a linear layer) estimates frames per phoneme. During training those targets come from the attention of an autoregressive Transformer TTS teacher. The length regulator repeats each 384-d state d_i times (69 frames here), and 6 more FFT blocks and a linear layer produce all 80-band mel frames in one pass, with no causal mask. A vocoder (WaveGlow in the paper) makes the waveform.
Hmel=LR(Hpho,D,α),Tmel=∑i=1N⌊α di⌉,Ldur=1N∑i(log⁡d^i−log⁡di)2\mathcal H_{\text{mel}}=\mathcal{LR}(\mathcal H_{\text{pho}},\mathcal D,\alpha),\qquad T_{\text{mel}}=\sum_{i=1}^{N}\big\lfloor\alpha\,d_i\big\rceil,\qquad \mathcal L_{\text{dur}}=\frac1N\sum_i\big(\log\hat d_i-\log d_i\big)^2
Hpho\mathcal H_{\text{pho}}
NN phoneme states from the encoder FFT blocks
D=(d1,…,dN)\mathcal D=(d_1,\dots,d_N)
durations in mel frames: teacher-extracted during training, predicted d^i\hat d_i at inference
α\alpha
speed control: α=1.5\alpha=1.5 is 1.5× slower, α=0.5\alpha=0.5 twice as fast
Ldur\mathcal L_{\text{dur}}
duration loss, MSE in the log domain; implementations add an offset such as log⁡(d+1)\log(d+1) so that d=0d=0 is allowed

Durations from a teacher's attention

There are no human labels for phoneme durations, so FastSpeech first trains an autoregressive Transformer TTS model (Li et al., 2019) as a teacher. Its encoder–decoder attention heads learn soft alignments between mel frames and phonemes. Not all heads are useful: some are diagonal, others are blurry or look at punctuation. FastSpeech chooses the head whose alignment is most diagonal, measured by the focus rate.

F=1S∑s=1Smax⁡1≤t≤Tas,t,dt=∑s=1S1[arg⁡max⁡t′as,t′=t]F=\frac1S\sum_{s=1}^{S}\max_{1\le t\le T}a_{s,t},\qquad d_t=\sum_{s=1}^{S}\mathbb 1\big[\arg\max_{t'}a_{s,t'}=t\big]
as,ta_{s,t}
teacher attention of mel frame ss on phoneme tt; each row sums to 1
S, TS,\ T
numbers of mel frames and phonemes
FF
focus rate: 1 for a hard one-hot alignment, 1/T1/T for uniform attention
dtd_t
duration of phoneme tt: the number of frames whose strongest attention is on it. ∑tdt=S\sum_t d_t=S

The teacher is used a second time for sequence-level Knowledge distillation. FastSpeech is trained on the mel spectrograms the teacher generates for each text, not on the recordings. The teacher's outputs are more consistent from one example to the next, which makes the one-to-many mapping from text to speech easier to fit for a model that predicts every frame independently.

Setup and results (Ren et al., 2019)

Data
LJSpeech: 13,100 clips, about 24 h, one female English speaker, 22,050 Hz; text converted to phonemes
Model
6 + 6 FFT blocks, hidden size 384, 2 attention heads, convolution kernel 3 with 1536 inner channels
Teacher
autoregressive Transformer TTS: gives the attention for durations and the distilled mel targets
Speed
about 270× faster mel generation and 38× faster end-to-end synthesis (with WaveGlow) than the Transformer TTS baseline
Robustness
no repeated or skipped words on the paper's set of 50 particularly hard sentences, where the autoregressive models made errors
Control
α from 0.5 to 1.5 changed speed smoothly; lengthening the space tokens added pauses between words

Extracting durations, live

teacher attention heads (phonemes × 69 frames) · focus rate F = mean over frames of max αhead 1 · F = 0.761 ★head 2 · F = 0.393head 3 · F = 0.650head 4 · F = 0.710head 1 (highest F, used by FastSpeech): dots mark each frame’s argmax phonemeFAE1STSPIY1CHdtrueα·dF 878AE1 101210S 999T 444S 888P 656IY1 151415CH 9109Σd = 69 frames (one argmax per frame) · |d − true| summed = 6 frames · 0 phonemes with d = 0FAE1STSPIY1CHLRα = 1.00 → 69 frames × 256 / 22050 Hz = 0.801 s, all generated in one parallel passthe autoregressive teacher needed 69 sequential decoder steps for the same 69 frames
teacher head
fig.Four toy teacher heads over "fast speech" (8 phonemes, 69 frames). The diagonal head 1 has the highest focus rate, F = 0.761, and is selected. Its argmax counts are 8, 10, 9, 4, 8, 6, 15, 9 against true durations of 7, 12, 9, 4, 8, 5, 14, 10, off by 6 frames in total. At α = 1 the length regulator gives 69 frames = 0.801 s at hop 256 and 22,050 Hz. α = 1.5 gives 105 frames (1.219 s) and α = 0.5 gives 36 frames (0.418 s), all in one parallel pass, where the teacher needed 69 sequential steps. Focus rate is only a heuristic. Head 4, which stares at the first and last phonemes, scores F = 0.710, second best, but gives 6 of 8 phonemes zero frames. The blurry head 2 (F = 0.393) is off by 28 frames.

Worked example

FrameSIY1Tmaxargmax
10.800.150.050.80S
20.600.300.100.60S
30.250.650.100.65IY1
40.100.800.100.80IY1
50.100.550.350.55IY1
60.050.300.650.65T
70.050.100.850.85T
One teacher head for "seat" (S IY1 T, 7 mel frames). Each row is one frame's attention.
  1. 1Focus rate: F=(0.80+0.60+0.65+0.80+0.55+0.65+0.85)/7=4.90/7=0.700F=(0.80+0.60+0.65+0.80+0.55+0.65+0.85)/7=4.90/7=0.700. A flatter head, 0.5a+0.5/30.5a+0.5/3, has the same argmaxes but F=0.5167F=0.5167, so this head is chosen.
  2. 2Durations from argmax counts: d=(2,3,2)d=(2,3,2), summing to S=7S=7.
  3. 3Length regulator at α=1.5\alpha=1.5: 1.5 d=(3,4.5,3)1.5\,d=(3,4.5,3), rounded half up to (3,5,3)(3,5,3), giving 11 frames. Banker's rounding, as in torch.round or np.round, would give 4 for IY1 and 10 frames in total.
  4. 4At α=0.5\alpha=0.5: (1,1.5,1)→(1,2,1)(1,1.5,1)\to(1,2,1), 4 frames. At hop 256 and 22,050 Hz, 7 frames are 81.3 ms of audio.
python
1import numpy as np
2
3def focus_rate(a): # a: (S frames, T phonemes), each row sums to 1
4 return a.max(axis=1).mean()
5
6def extract_durations(heads): # heads: (H, S, T) teacher encoder-decoder attention
7 best = int(np.argmax([focus_rate(a) for a in heads]))
8 d = np.bincount(heads[best].argmax(axis=1), minlength=heads.shape[2])
9 return best, d # d sums to S: every frame is given to one phoneme
10
11def length_regulator(h, d, alpha=1.0): # h: (T, dim) phoneme states from the encoder
12 reps = np.floor(alpha * d + 0.5).astype(int) # round half up
13 return np.repeat(h, reps, axis=0) # (sum(reps), dim)
14
15# teacher head for "seat" = S IY1 T, 7 mel frames
16sharp = np.array([[.80, .15, .05], [.60, .30, .10], [.25, .65, .10], [.10, .80, .10],
17 [.10, .55, .35], [.05, .30, .65], [.05, .10, .85]])
18blurry = 0.5 * sharp + 0.5 / 3 # same argmaxes, flatter rows
19best, d = extract_durations(np.stack([blurry, sharp]))
20print(best, d, round(focus_rate(sharp), 4), round(focus_rate(blurry), 4)) # 1 [2 3 2] 0.7 0.5167
21
22h = np.random.default_rng(0).normal(size=(3, 384))
23for alpha in (0.5, 1.0, 1.5):
24 print(alpha, length_regulator(h, d, alpha).shape) # (4, 384) (7, 384) (11, 384)
Duration extraction and the length regulator in numpy (run; outputs as shown)

Use when

  • +Parallel mel generation: latency barely grows with utterance length on a GPU
  • +No skipped or repeated words, since alignment is explicit and monotonic
  • +Speed and pauses are controllable through α and the durations of space tokens

Avoid when

  • −Needs a trained autoregressive teacher, which makes the training pipeline longer
  • −Duration and mel quality are bounded by the teacher, and distillation loses some variability
  • −Durations are deterministic, so prosody is averaged; pitch and energy are not modelled explicitly

Check yourself

A teacher head assigns all 120 frames of an utterance either to the first or to the last phoneme, each with weight 0.9. What is its focus rate, and is it a good head for durations?

Check yourself

FastSpeech predicts durations summing to 200 frames (hop 256, 22,050 Hz). How many sequential decoder steps does the mel decoder need, and how long is the audio at α = 1?

Remember

  • ✓Feed-forward Transformer: 6 FFT blocks on phonemes, a length regulator, 6 FFT blocks on frames; self-attention plus 1-D convolution, hidden size 384.
  • ✓Durations come from the teacher head with the highest focus rate F=1S∑smax⁡tas,tF=\frac1S\sum_s\max_t a_{s,t}; dtd_t is the number of frames whose argmax is tt.
  • ✓The duration predictor (two convolutions and a linear layer) is trained with MSE on log durations and used at inference; T=∑i⌊αdi⌉T=\sum_i\lfloor\alpha d_i\rceil.
  • ✓Sequence-level distillation: train on the teacher's generated mels. About 270× faster mel generation and 38× faster end-to-end synthesis.
  • ✓Explicit monotonic alignment: no skipping or repeating; α gives speed control. FastSpeech 2 removes the teacher.
← back to Deep Learning roadmap