# FastSpeech

> Ren et al. (2019): a non-autoregressive feed-forward Transformer copies each phoneme state for its predicted duration (length regulator, speed α) and emits all mel frames at once. Durations come from a teacher's attention; 270× faster mel generation.

Source: https://www.mlfoundry.in/learn/fastspeech  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**FastSpeech:** is a non-autoregressive text-to-speech acoustic model. It predicts how many mel frames each phoneme lasts, expands the phoneme sequence to that length, and generates every mel frame in parallel. It was introduced by Ren, Ruan, Tan, Qin, Zhao, Zhao and Liu in "FastSpeech: Fast, Robust and Controllable Text to Speech" (NeurIPS 2019). The phoneme durations are extracted from the attention of an autoregressive teacher model.

Autoregressive models such as **Tacotron 2** generate mel frames one at a time, each conditioned on the last. A 5-second utterance at a 12.5 ms hop takes 400 sequential decoder steps, so latency grows with length. Their soft attention can also skip or repeat words, and they give no direct handle on speaking rate.

FastSpeech removes the attention between text and speech. It makes the alignment explicit: phoneme $i$ gets exactly $d_i$ frames, in order. Once the frame count is known, a feed-forward network can produce all frames at once. The length regulator itself is introduced in **Text-to-speech**. This article covers where the durations come from and what parallel generation buys.

## Architecture

The model is built from **feed-forward Transformer (FFT) blocks**. Each block is multi-head **Self-attention** followed by a two-layer 1-D convolution (kernel 3) in place of the position-wise feed-forward layer of a **Transformer encoder**, with residual connections and layer normalisation. The convolution suits speech, where neighbouring frames and phonemes are closely related. Six FFT blocks run on the phoneme side and six on the mel side, with hidden size 384 and 2 attention heads.

> Figure: FastSpeech on "fast speech". The 8 phonemes pass through 6 FFT blocks. A duration predictor (two 1-D convolutions and a linear layer) estimates frames per phoneme. During training those targets come from the attention of an autoregressive Transformer TTS teacher. The length regulator repeats each 384-d state d_i times (69 frames here), and 6 more FFT blocks and a linear layer produce all 80-band mel frames in one pass, with no causal mask. A vocoder (WaveGlow in the paper) makes the waveform.

$$
\mathcal H_{\text{mel}}=\mathcal{LR}(\mathcal H_{\text{pho}},\mathcal D,\alpha),\qquad T_{\text{mel}}=\sum_{i=1}^{N}\big\lfloor\alpha\,d_i\big\rceil,\qquad \mathcal L_{\text{dur}}=\frac1N\sum_i\big(\log\hat d_i-\log d_i\big)^2
$$

- $\mathcal H_{\text{pho}}$: $N$ phoneme states from the encoder FFT blocks
- $\mathcal D=(d_1,\dots,d_N)$: durations in mel frames: teacher-extracted during training, predicted $\hat d_i$ at inference
- $\alpha$: speed control: $\alpha=1.5$ is 1.5× slower, $\alpha=0.5$ twice as fast
- $\mathcal L_{\text{dur}}$: duration loss, MSE in the log domain; implementations add an offset such as $\log(d+1)$ so that $d=0$ is allowed

## Durations from a teacher's attention

There are no human labels for phoneme durations, so FastSpeech first trains an autoregressive Transformer TTS model (Li et al., 2019) as a teacher. Its encoder–decoder attention heads learn soft alignments between mel frames and phonemes. Not all heads are useful: some are diagonal, others are blurry or look at punctuation. FastSpeech chooses the head whose alignment is most diagonal, measured by the **focus rate**.

$$
F=\frac1S\sum_{s=1}^{S}\max_{1\le t\le T}a_{s,t},\qquad d_t=\sum_{s=1}^{S}\mathbb 1\big[\arg\max_{t'}a_{s,t'}=t\big]
$$

- $a_{s,t}$: teacher attention of mel frame $s$ on phoneme $t$; each row sums to 1
- $S,\ T$: numbers of mel frames and phonemes
- $F$: focus rate: 1 for a hard one-hot alignment, $1/T$ for uniform attention
- $d_t$: duration of phoneme $t$: the number of frames whose strongest attention is on it. $\sum_t d_t=S$

The teacher is used a second time for **sequence-level Knowledge distillation**. FastSpeech is trained on the mel spectrograms the teacher generates for each text, not on the recordings. The teacher's outputs are more consistent from one example to the next, which makes the one-to-many mapping from text to speech easier to fit for a model that predicts every frame independently.

**Setup and results (Ren et al., 2019)**

- **Data:** LJSpeech: 13,100 clips, about 24 h, one female English speaker, 22,050 Hz; text converted to phonemes
- **Model:** 6 + 6 FFT blocks, hidden size 384, 2 attention heads, convolution kernel 3 with 1536 inner channels
- **Teacher:** autoregressive Transformer TTS: gives the attention for durations and the distilled mel targets
- **Speed:** about 270× faster mel generation and 38× faster end-to-end synthesis (with WaveGlow) than the Transformer TTS baseline
- **Robustness:** no repeated or skipped words on the paper's set of 50 particularly hard sentences, where the autoregressive models made errors
- **Control:** α from 0.5 to 1.5 changed speed smoothly; lengthening the space tokens added pauses between words

## Extracting durations, live

> Figure: Four toy teacher heads over "fast speech" (8 phonemes, 69 frames). The diagonal head 1 has the highest focus rate, F = 0.761, and is selected. Its argmax counts are 8, 10, 9, 4, 8, 6, 15, 9 against true durations of 7, 12, 9, 4, 8, 5, 14, 10, off by 6 frames in total. At α = 1 the length regulator gives 69 frames = 0.801 s at hop 256 and 22,050 Hz. α = 1.5 gives 105 frames (1.219 s) and α = 0.5 gives 36 frames (0.418 s), all in one parallel pass, where the teacher needed 69 sequential steps. Focus rate is only a heuristic. Head 4, which stares at the first and last phonemes, scores F = 0.710, second best, but gives 6 of 8 phonemes zero frames. The blurry head 2 (F = 0.393) is off by 28 frames.

## Worked example

One teacher head for "seat" (S IY1 T, 7 mel frames). Each row is one frame's attention.

| Frame | S | IY1 | T | max | argmax |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.80 | 0.15 | 0.05 | 0.80 | S |
| 2 | 0.60 | 0.30 | 0.10 | 0.60 | S |
| 3 | 0.25 | 0.65 | 0.10 | 0.65 | IY1 |
| 4 | 0.10 | 0.80 | 0.10 | 0.80 | IY1 |
| 5 | 0.10 | 0.55 | 0.35 | 0.55 | IY1 |
| 6 | 0.05 | 0.30 | 0.65 | 0.65 | T |
| 7 | 0.05 | 0.10 | 0.85 | 0.85 | T |

1. Focus rate: $F=(0.80+0.60+0.65+0.80+0.55+0.65+0.85)/7=4.90/7=0.700$. A flatter head, $0.5a+0.5/3$, has the same argmaxes but $F=0.5167$, so this head is chosen.
2. Durations from argmax counts: $d=(2,3,2)$, summing to $S=7$.
3. Length regulator at $\alpha=1.5$: $1.5\,d=(3,4.5,3)$, rounded half up to $(3,5,3)$, giving 11 frames. Banker's rounding, as in `torch.round` or `np.round`, would give 4 for IY1 and 10 frames in total.
4. At $\alpha=0.5$: $(1,1.5,1)\to(1,2,1)$, 4 frames. At hop 256 and 22,050 Hz, 7 frames are 81.3 ms of audio.

```python
import numpy as np

def focus_rate(a):                      # a: (S frames, T phonemes), each row sums to 1
    return a.max(axis=1).mean()

def extract_durations(heads):           # heads: (H, S, T) teacher encoder-decoder attention
    best = int(np.argmax([focus_rate(a) for a in heads]))
    d = np.bincount(heads[best].argmax(axis=1), minlength=heads.shape[2])
    return best, d                      # d sums to S: every frame is given to one phoneme

def length_regulator(h, d, alpha=1.0):  # h: (T, dim) phoneme states from the encoder
    reps = np.floor(alpha * d + 0.5).astype(int)   # round half up
    return np.repeat(h, reps, axis=0)              # (sum(reps), dim)

# teacher head for "seat" = S IY1 T, 7 mel frames
sharp = np.array([[.80, .15, .05], [.60, .30, .10], [.25, .65, .10], [.10, .80, .10],
                  [.10, .55, .35], [.05, .30, .65], [.05, .10, .85]])
blurry = 0.5 * sharp + 0.5 / 3          # same argmaxes, flatter rows
best, d = extract_durations(np.stack([blurry, sharp]))
print(best, d, round(focus_rate(sharp), 4), round(focus_rate(blurry), 4))   # 1 [2 3 2] 0.7 0.5167

h = np.random.default_rng(0).normal(size=(3, 384))
for alpha in (0.5, 1.0, 1.5):
    print(alpha, length_regulator(h, d, alpha).shape)   # (4, 384) (7, 384) (11, 384)
```

Duration extraction and the length regulator in numpy (run; outputs as shown)

> **The teacher limits the student:** Durations from argmax counts are only as good as the chosen attention head. They are coarse, may assign a phoneme zero frames, and a high focus rate does not prove a head is correct. Distilled mel targets are also smoother than real speech, which can make the output over-smoothed. **FastSpeech 2** drops the teacher: it takes durations from a forced aligner, trains on real mel spectrograms, and adds pitch and energy predictors.

> **Why parallel decoding is robust:** In an attention decoder, a single bad alignment step feeds wrong context into every later step. With a length regulator, the alignment is fixed before decoding starts: every phoneme with $d_i\ge1$ is spoken exactly once and in order. Errors can still occur in the durations or the audio quality, but words cannot be skipped or repeated by the decoder.

**Strengths**

- Parallel mel generation: latency barely grows with utterance length on a GPU
- No skipped or repeated words, since alignment is explicit and monotonic
- Speed and pauses are controllable through α and the durations of space tokens

**Limitations**

- Needs a trained autoregressive teacher, which makes the training pipeline longer
- Duration and mel quality are bounded by the teacher, and distillation loses some variability
- Durations are deterministic, so prosody is averaged; pitch and energy are not modelled explicitly

**Question:** A teacher head assigns all 120 frames of an utterance either to the first or to the last phoneme, each with weight 0.9. What is its focus rate, and is it a good head for durations?

- A. 0.9; yes, because the focus rate is high
- B. 0.9; no, because it gives every middle phoneme zero frames, so focus rate alone can be misleading
- C. 0.5; no
- D. 1.0; yes

**Answer:** B. 0.9; no, because it gives every middle phoneme zero frames, so focus rate alone can be misleading — The focus rate averages each frame's maximum weight, so $F=0.9$. That rewards sharpness, not a correct diagonal. Here every middle phoneme gets $d=0$. The paper relies on the most-focused head in practice being diagonal; the extracted durations should still be checked.

**Question:** FastSpeech predicts durations summing to 200 frames (hop 256, 22,050 Hz). How many sequential decoder steps does the mel decoder need, and how long is the audio at α = 1?

- A. 200 steps; 2.32 s
- B. One parallel pass through the mel-side FFT blocks; 2.32 s
- C. One parallel pass; 0.20 s
- D. 200 steps; 0.20 s

**Answer:** B. One parallel pass through the mel-side FFT blocks; 2.32 s — All 200 frame positions are filled by the length regulator and processed together by self-attention and convolution, so there are no sequential frame steps. The audio is $200\times256/22{,}050=2.322$ s.

**Key points**

- Feed-forward Transformer: 6 FFT blocks on phonemes, a length regulator, 6 FFT blocks on frames; self-attention plus 1-D convolution, hidden size 384.
- Durations come from the teacher head with the highest focus rate $F=\frac1S\sum_s\max_t a_{s,t}$; $d_t$ is the number of frames whose argmax is $t$.
- The duration predictor (two convolutions and a linear layer) is trained with MSE on log durations and used at inference; $T=\sum_i\lfloor\alpha d_i\rceil$.
- Sequence-level distillation: train on the teacher's generated mels. About 270× faster mel generation and 38× faster end-to-end synthesis.
- Explicit monotonic alignment: no skipping or repeating; α gives speed control. FastSpeech 2 removes the teacher.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
