# Text-to-speech

> Text → normalised words → phonemes → mel spectrogram (Tacotron 2, FastSpeech 2) → waveform (WaveNet, HiFi-GAN). VITS does this end to end, VALL-E predicts codec tokens, and listeners rate quality as MOS.

Source: https://www.mlfoundry.in/learn/text-to-speech  
Section: Deep Learning › Deep Learning for NLP  
Level: advanced · 8 min read · updated 2026-10-15

**Text-to-speech (TTS, speech synthesis):** generates a waveform that speaks a given text, usually in a chosen voice. Neural TTS is typically split in two. An **acoustic model** maps phonemes to a mel spectrogram: Tacotron 2 (Shen et al., 2018) or FastSpeech 2 (Ren et al., 2021). A **vocoder** turns the mel into audio: WaveNet (van den Oord et al., 2016) or HiFi-GAN (Kong, Kim and Bae, 2020). End-to-end models such as VITS (Kim, Kong and Son, 2021) and codec language models such as VALL-E (Wang et al., 2023) merge or replace these stages.

TTS is speech-to-text run backwards, and the inverse is harder in one respect. The output is far longer than the input: one second of 22.05 kHz audio is 22,050 samples, made from about a dozen phonemes. The output is also one-to-many. The same sentence can be spoken with any duration, pitch contour, emphasis or voice, and all of them are "correct".

Before neural models, systems either concatenated recorded units chosen from a large database (unit selection) or generated vocoder parameters from HMMs (statistical parametric synthesis). Unit selection sounded natural but glitched at joins. Parametric synthesis was smooth but buzzy and muffled.

## The pipeline

**Text normalisation** rewrites non-standard words: "Dr." → "doctor", "$5" → "five dollars", "1990" → "nineteen ninety". **Grapheme-to-phoneme (G2P)** conversion looks words up in a lexicon such as CMUdict and predicts the rest with a model. The result is a phoneme sequence with stress marks, e.g. HH AH0 L OW1 for "hello". Both steps are language-specific, and their errors cannot be fixed by later stages.

> Figure: Two-stage neural TTS on "Dr. Lee paid $5.": normalisation, G2P into 18 ARPAbet phonemes, an acoustic model (autoregressive Tacotron 2 or parallel FastSpeech 2) predicting an 80-band log-mel, and a vocoder producing 256 samples per mel frame. The dashed boxes are the single-model alternatives: VITS (text to waveform) and VALL-E (a language model over neural-codec tokens).

**Tacotron 2** encodes characters or phonemes with convolutions and a BiLSTM. A location-sensitive attention decoder then emits one 80-band mel frame per step, with a stop token deciding when to end. Its mel used 50 ms windows and a 12.5 ms hop. With a WaveNet vocoder it reached a MOS of 4.53, against 4.58 for professional recordings. Because it is autoregressive and attention-based, it is slow, and it can skip or repeat words when attention fails.

**FastSpeech** (Ren et al., 2019) removes both problems. It predicts an integer **duration** $d_i$ (in frames) for every phoneme. A **length regulator** copies each phoneme's hidden state $d_i$ times, and a non-autoregressive decoder produces all frames at once. The paper reported 270× faster mel generation and 38× faster end-to-end synthesis than its autoregressive baseline. **FastSpeech 2** trains on durations from a forced aligner and adds pitch and energy predictors (the "variance adaptor").

$$
\mathrm{LR}(h_{1:N},d,\alpha)=\big[\underbrace{h_1,\dots,h_1}_{\lfloor\alpha d_1\rceil},\;\dots,\;\underbrace{h_N,\dots,h_N}_{\lfloor\alpha d_N\rceil}\big],\qquad T=\sum_{i=1}^{N}\lfloor\alpha d_i\rceil
$$

- $h_i$: encoder state of phoneme $i$
- $d_i$: predicted duration in mel frames; FastSpeech 2 trains the duration predictor with MSE on log-durations
- $\alpha$: speed control: $\alpha>1$ slows speech down, $\alpha<1$ speeds it up
- $T$: number of mel frames; audio length is $T\cdot\text{hop}$ samples

**Vocoders.** WaveNet generates one sample at a time with stacks of dilated causal convolutions (see **Autoregressive generative models**). It produces excellent audio but is very slow. HiFi-GAN upsamples the mel by the hop length (×256) with transposed convolutions and is trained as a **Generative adversarial network (GAN)**. Its discriminators look at the waveform reshaped by periods 2, 3, 5, 7 and 11 (multi-period) and at several downsampled scales (multi-scale), plus a mel L1 loss. It runs far faster than real time on a GPU.

**VITS** trains one model from phonemes to waveform. It is a conditional VAE whose prior is enriched with a **Normalizing flows** stack and whose decoder is a HiFi-GAN-style generator trained adversarially. Monotonic alignment search finds durations during training, with no external aligner. A stochastic duration predictor lets the same text come out with different rhythms. **VALL-E** treats TTS as language modelling. Speech is encoded by the EnCodec neural codec into 8 codebooks of discrete tokens (see **Vector-quantized VAE (VQ-VAE)**). A decoder-only Transformer predicts the first codebook autoregressively from phonemes plus the tokens of a 3-second prompt of an unseen speaker, and a non-autoregressive model fills in the other 7. It was trained on 60,000 hours of LibriLight.

## Data and evaluation

**Benchmarks and metrics**

- **LJSpeech:** 13,100 clips, about 24 h, one female English speaker, 22,050 Hz; the standard single-speaker benchmark
- **Typical features:** 80-band log-mel; LJSpeech recipes use n_fft 1024 and hop 256 (11.6 ms per frame)
- **MOS:** mean opinion score: listeners rate naturalness 1 (bad) to 5 (excellent); report the mean with a 95% confidence interval, $\pm1.96\,s/\sqrt{n}$
- **Also used:** CMOS (side-by-side comparison), WER of an ASR model on the output (intelligibility), speaker-embedding cosine similarity (voice cloning)

## Durations and mel filters, live

> Figure: Top: the length regulator on "hello world" (8 phonemes). At α = 1 the predicted durations sum to 63 frames × 256 = 16,128 samples = 0.731 s. At α = 2 the same phonemes fill 126 frames (1.463 s). At α = 0.5 they fill 33 frames, not 31.5, because each duration is rounded separately (half up here). Applying α = 1.5 to "world" only gives 81 frames (0.940 s), which is how per-word emphasis is controlled. Bottom: the 80-filter mel bank with n_fft = 1024. Filter 1 is centred at 22.1 Hz and is 44.9 Hz wide, filter 80 at 7733.5 Hz is 524.8 Hz wide (11.7×), and the narrowest filter covers 2 FFT bins. With n_fft = 512 and 128 mels, 5 filters are empty.

## Worked example

Worked example: length regulator on "hello world" at α = 1.25 (22,050 Hz, hop 256)

| Phoneme | HH | AH0 | L | OW1 | W | ER1 | L | D | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| predicted d | 5 | 4 | 6 | 14 | 6 | 12 | 7 | 9 | 63 |
| 1.25 · d | 6.25 | 5 | 7.5 | 17.5 | 7.5 | 15 | 8.75 | 11.25 | 78.75 |
| rounded | 6 | 5 | 8 | 18 | 8 | 15 | 9 | 11 | 80 |

So the decoder produces $T=80$ mel frames, i.e. $80\times256=20{,}480$ samples $=0.929$ s, against 0.731 s at $\alpha=1$. Rounding added $1.25$ frames over the exact $78.75$. The stressed vowels OW1 and ER1 hold 33 of the 80 frames.

```python
import torch

def length_regulator(h, d_hat, alpha=1.0):
    """h: (N, d_model) phoneme states; d_hat: (N,) predicted durations in frames."""
    d = torch.clamp(torch.round(d_hat * alpha), min=1).long()
    return torch.repeat_interleave(h, d, dim=0), d       # (sum(d), d_model)

h = torch.randn(8, 256)                                   # HH AH0 L OW1 W ER1 L D
d_hat = torch.tensor([5., 4, 6, 14, 6, 12, 7, 9])
frames, d = length_regulator(h, d_hat, alpha=1.25)
print(d.tolist(), frames.shape)   # [6, 5, 8, 18, 8, 15, 9, 11] torch.Size([80, 256])
print(frames.shape[0] * 256 / 22050)                      # 0.9288 s of audio
```

A FastSpeech-style length regulator in PyTorch (not run here; the frame counts were checked separately)

> **Mel settings must match end to end:** A vocoder learns to invert one exact mel definition: sample rate, n_fft, hop, number of bands, frequency range, log base and normalisation. Feeding it mels from an acoustic model trained with different settings gives noise or metallic audio, even if both use "80-band mel". Check the configs before pairing a pretrained acoustic model with a pretrained vocoder.

> **Why the two-stage split works:** The mel spectrogram is about 256× shorter than the waveform and discards phase, so the acoustic model handles the hard one-to-many choices (duration, prosody) at a low frame rate. The vocoder only has to solve a local, nearly deterministic problem: filling in phase and fine detail. This is also why vocoders transfer well across speakers.

- **Assistants and accessibility:** Screen readers, voice assistants and navigation prompts, where FastSpeech-style parallel models keep latency low.
- **Voice cloning and dubbing:** Zero-shot speaker adaptation from a few seconds of audio (VALL-E style prompts); this raises consent and spoofing concerns.
- **Data for ASR:** Synthetic speech augments training data for **Speech-to-text**, especially for rare words and names.

**Question:** A FastSpeech model predicts durations that sum to 200 frames for a sentence (hop 256, 22,050 Hz). With α = 0.8, about how long is the audio?

- A. about 1.86 s
- B. about 2.32 s
- C. about 2.90 s
- D. exactly 0.8 s

**Answer:** A. about 1.86 s — $0.8\times200=160$ frames (up to per-phoneme rounding), and $160\times256/22{,}050=1.858$ s. At $\alpha=1$ it would be $2.322$ s.

**Question:** Why can Tacotron 2 occasionally skip or repeat words while FastSpeech 2 cannot?

- A. Tacotron 2 uses a smaller vocabulary
- B. Tacotron 2's soft attention learns its own text–frame alignment and can jump; FastSpeech 2 fixes the alignment with explicit durations
- C. FastSpeech 2 uses a better vocoder
- D. Tacotron 2 has no encoder

**Answer:** B. Tacotron 2's soft attention learns its own text–frame alignment and can jump; FastSpeech 2 fixes the alignment with explicit durations — The length regulator gives every phoneme exactly $d_i\ge1$ frames in order, so each phoneme is spoken once. An autoregressive attention decoder can attend to the wrong position and stop early or loop.

**Key points**

- Pipeline: normalise text → G2P phonemes → acoustic model → 80-band mel → vocoder → waveform.
- Length regulator: repeat $h_i$ for $\lfloor\alpha d_i\rceil$ frames; $T=\sum_i\lfloor\alpha d_i\rceil$ frames, audio $=T\cdot\text{hop}$ samples.
- Tacotron 2 is autoregressive with attention (MOS 4.53 vs 4.58 human); FastSpeech 2 is parallel with duration, pitch and energy predictors.
- Vocoders: WaveNet (autoregressive, slow) → HiFi-GAN (GAN, multi-period discriminator with periods 2, 3, 5, 7, 11, faster than real time).
- VITS: one VAE + flow + GAN model; VALL-E: a language model over EnCodec tokens with a 3 s speaker prompt. Quality is judged by MOS with confidence intervals.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
