Definition
Text-to-speech (TTS, speech synthesis) generates a waveform that speaks a given text, usually in a chosen voice. Neural TTS is typically split in two. An acoustic model maps phonemes to a mel spectrogram: Tacotron 2 (Shen et al., 2018) or FastSpeech 2 (Ren et al., 2021). A vocoder turns the mel into audio: WaveNet (van den Oord et al., 2016) or HiFi-GAN (Kong, Kim and Bae, 2020). End-to-end models such as VITS (Kim, Kong and Son, 2021) and codec language models such as VALL-E (Wang et al., 2023) merge or replace these stages.
TTS is speech-to-text run backwards, and the inverse is harder in one respect. The output is far longer than the input: one second of 22.05 kHz audio is 22,050 samples, made from about a dozen phonemes. The output is also one-to-many. The same sentence can be spoken with any duration, pitch contour, emphasis or voice, and all of them are "correct".
Before neural models, systems either concatenated recorded units chosen from a large database (unit selection) or generated vocoder parameters from HMMs (statistical parametric synthesis). Unit selection sounded natural but glitched at joins. Parametric synthesis was smooth but buzzy and muffled.
The pipeline
Text normalisation rewrites non-standard words: "Dr." → "doctor", "$5" → "five dollars", "1990" → "nineteen ninety". Grapheme-to-phoneme (G2P) conversion looks words up in a lexicon such as CMUdict and predicts the rest with a model. The result is a phoneme sequence with stress marks, e.g. HH AH0 L OW1 for "hello". Both steps are language-specific, and their errors cannot be fixed by later stages.
Tacotron 2 encodes characters or phonemes with convolutions and a BiLSTM. A location-sensitive attention decoder then emits one 80-band mel frame per step, with a stop token deciding when to end. Its mel used 50 ms windows and a 12.5 ms hop. With a WaveNet vocoder it reached a MOS of 4.53, against 4.58 for professional recordings. Because it is autoregressive and attention-based, it is slow, and it can skip or repeat words when attention fails.
FastSpeech (Ren et al., 2019) removes both problems. It predicts an integer duration (in frames) for every phoneme. A length regulator copies each phoneme's hidden state times, and a non-autoregressive decoder produces all frames at once. The paper reported 270× faster mel generation and 38× faster end-to-end synthesis than its autoregressive baseline. FastSpeech 2 trains on durations from a forced aligner and adds pitch and energy predictors (the "variance adaptor").
- encoder state of phoneme
- predicted duration in mel frames; FastSpeech 2 trains the duration predictor with MSE on log-durations
- speed control: slows speech down, speeds it up
- number of mel frames; audio length is samples
Vocoders. WaveNet generates one sample at a time with stacks of dilated causal convolutions (see Autoregressive generative models). It produces excellent audio but is very slow. HiFi-GAN upsamples the mel by the hop length (×256) with transposed convolutions and is trained as a Generative adversarial network (GAN). Its discriminators look at the waveform reshaped by periods 2, 3, 5, 7 and 11 (multi-period) and at several downsampled scales (multi-scale), plus a mel L1 loss. It runs far faster than real time on a GPU.
VITS trains one model from phonemes to waveform. It is a conditional VAE whose prior is enriched with a Normalizing flows stack and whose decoder is a HiFi-GAN-style generator trained adversarially. Monotonic alignment search finds durations during training, with no external aligner. A stochastic duration predictor lets the same text come out with different rhythms. VALL-E treats TTS as language modelling. Speech is encoded by the EnCodec neural codec into 8 codebooks of discrete tokens (see Vector-quantized VAE (VQ-VAE)). A decoder-only Transformer predicts the first codebook autoregressively from phonemes plus the tokens of a 3-second prompt of an unseen speaker, and a non-autoregressive model fills in the other 7. It was trained on 60,000 hours of LibriLight.
Data and evaluation
Benchmarks and metrics
- LJSpeech
- 13,100 clips, about 24 h, one female English speaker, 22,050 Hz; the standard single-speaker benchmark
- Typical features
- 80-band log-mel; LJSpeech recipes use n_fft 1024 and hop 256 (11.6 ms per frame)
- MOS
- mean opinion score: listeners rate naturalness 1 (bad) to 5 (excellent); report the mean with a 95% confidence interval,
- Also used
- CMOS (side-by-side comparison), WER of an ASR model on the output (intelligibility), speaker-embedding cosine similarity (voice cloning)
Durations and mel filters, live
Worked example
| Phoneme | HH | AH0 | L | OW1 | W | ER1 | L | D | Total |
|---|---|---|---|---|---|---|---|---|---|
| predicted d | 5 | 4 | 6 | 14 | 6 | 12 | 7 | 9 | 63 |
| 1.25 · d | 6.25 | 5 | 7.5 | 17.5 | 7.5 | 15 | 8.75 | 11.25 | 78.75 |
| rounded | 6 | 5 | 8 | 18 | 8 | 15 | 9 | 11 | 80 |
So the decoder produces mel frames, i.e. samples s, against 0.731 s at . Rounding added frames over the exact . The stressed vowels OW1 and ER1 hold 33 of the 80 frames.
1import torch23def length_regulator(h, d_hat, alpha=1.0):4 """h: (N, d_model) phoneme states; d_hat: (N,) predicted durations in frames."""5 d = torch.clamp(torch.round(d_hat * alpha), min=1).long()6 return torch.repeat_interleave(h, d, dim=0), d # (sum(d), d_model)78h = torch.randn(8, 256) # HH AH0 L OW1 W ER1 L D9d_hat = torch.tensor([5., 4, 6, 14, 6, 12, 7, 9])10frames, d = length_regulator(h, d_hat, alpha=1.25)11print(d.tolist(), frames.shape) # [6, 5, 8, 18, 8, 15, 9, 11] torch.Size([80, 256])12print(frames.shape[0] * 256 / 22050) # 0.9288 s of audioWhere it's used
Assistants and accessibility
Screen readers, voice assistants and navigation prompts, where FastSpeech-style parallel models keep latency low.
Voice cloning and dubbing
Zero-shot speaker adaptation from a few seconds of audio (VALL-E style prompts); this raises consent and spoofing concerns.
Data for ASR
Synthetic speech augments training data for Speech-to-text, especially for rare words and names.
Check yourself
A FastSpeech model predicts durations that sum to 200 frames for a sentence (hop 256, 22,050 Hz). With α = 0.8, about how long is the audio?
Check yourself
Why can Tacotron 2 occasionally skip or repeat words while FastSpeech 2 cannot?
Remember
- ✓Pipeline: normalise text → G2P phonemes → acoustic model → 80-band mel → vocoder → waveform.
- ✓Length regulator: repeat for frames; frames, audio samples.
- ✓Tacotron 2 is autoregressive with attention (MOS 4.53 vs 4.58 human); FastSpeech 2 is parallel with duration, pitch and energy predictors.
- ✓Vocoders: WaveNet (autoregressive, slow) → HiFi-GAN (GAN, multi-period discriminator with periods 2, 3, 5, 7, 11, faster than real time).
- ✓VITS: one VAE + flow + GAN model; VALL-E: a language model over EnCodec tokens with a 3 s speaker prompt. Quality is judged by MOS with confidence intervals.