Text-to-speech

Text → normalised words → phonemes → mel spectrogram (Tacotron 2, FastSpeech 2) → waveform (WaveNet, HiFi-GAN). VITS does this end to end, VALL-E predicts codec tokens, and listeners rate quality as MOS.

8 min read858 wordsadvanced0 viewsupdated 2026-10-15

Definition

Text-to-speech (TTS, speech synthesis) generates a waveform that speaks a given text, usually in a chosen voice. Neural TTS is typically split in two. An acoustic model maps phonemes to a mel spectrogram: Tacotron 2 (Shen et al., 2018) or FastSpeech 2 (Ren et al., 2021). A vocoder turns the mel into audio: WaveNet (van den Oord et al., 2016) or HiFi-GAN (Kong, Kim and Bae, 2020). End-to-end models such as VITS (Kim, Kong and Son, 2021) and codec language models such as VALL-E (Wang et al., 2023) merge or replace these stages.

TTS is speech-to-text run backwards, and the inverse is harder in one respect. The output is far longer than the input: one second of 22.05 kHz audio is 22,050 samples, made from about a dozen phonemes. The output is also one-to-many. The same sentence can be spoken with any duration, pitch contour, emphasis or voice, and all of them are "correct".

Before neural models, systems either concatenated recorded units chosen from a large database (unit selection) or generated vocoder parameters from HMMs (statistical parametric synthesis). Unit selection sounded natural but glitched at joins. Parametric synthesis was smooth but buzzy and muffled.

The pipeline

Text normalisation rewrites non-standard words: "Dr." → "doctor", "$5" → "five dollars", "1990" → "nineteen ninety". Grapheme-to-phoneme (G2P) conversion looks words up in a lexicon such as CMUdict and predicts the rest with a model. The result is a phoneme sequence with stress marks, e.g. HH AH0 L OW1 for "hello". Both steps are language-specific, and their errors cannot be fixed by later stages.

front end (language-specific) → acoustic model → vocoderinput textDr. Lee paid $5.abbrev., digits, $text normalisationdoctor lee paidfive dollarsrules / WFST / seq2seqG2P → phonemes (ARPAbet)D AA1 K T ER0 · L IY1 ·P EY1 D · F AY1 V ·D AA1 L ER0 Z (18 phones)Tacotron 2 (Shen et al. 2018)encoder: conv + BiLSTMlocation-sensitive attentionautoregressive: 1 mel frame per step+ stop token decides the lengthslow, can skip or repeat wordsFastSpeech 2 (Ren et al. 2021)phoneme encoder (FFT blocks)variance adaptor: duration, pitch,energy; length regulator repeatsh_i d_i times → Σd_i framesparallel decoder: all frames at oncelog-mel 80 × T framese.g. hop 256 @ 22.05 kHzvocoder: mel → waveformWaveNet (van den Oord et al. 2016):dilated causal conv, 1 sample/stepHiFi-GAN (Kong et al. 2020): convupsampler ×256, GAN with multi-period (2,3,5,7,11) + multi-scale DwaveformT × 256 samplesVITS (Kim et al. 2021): one modelphonemes → waveform: VAE + flow +GAN, monotonic alignment search,stochastic duration predictorVALL-E (Wang et al. 2023)LM over EnCodec tokens (8 codebooks)prompted by 3 s of a new speaker;codec decoder replaces the vocoder
fig.Two-stage neural TTS on "Dr. Lee paid $5.": normalisation, G2P into 18 ARPAbet phonemes, an acoustic model (autoregressive Tacotron 2 or parallel FastSpeech 2) predicting an 80-band log-mel, and a vocoder producing 256 samples per mel frame. The dashed boxes are the single-model alternatives: VITS (text to waveform) and VALL-E (a language model over neural-codec tokens).

Tacotron 2 encodes characters or phonemes with convolutions and a BiLSTM. A location-sensitive attention decoder then emits one 80-band mel frame per step, with a stop token deciding when to end. Its mel used 50 ms windows and a 12.5 ms hop. With a WaveNet vocoder it reached a MOS of 4.53, against 4.58 for professional recordings. Because it is autoregressive and attention-based, it is slow, and it can skip or repeat words when attention fails.

FastSpeech (Ren et al., 2019) removes both problems. It predicts an integer duration did_i (in frames) for every phoneme. A length regulator copies each phoneme's hidden state did_i times, and a non-autoregressive decoder produces all frames at once. The paper reported 270× faster mel generation and 38× faster end-to-end synthesis than its autoregressive baseline. FastSpeech 2 trains on durations from a forced aligner and adds pitch and energy predictors (the "variance adaptor").

LR(h1:N,d,α)=[h1,…,h1⏟⌊αd1⌉,  …,  hN,…,hN⏟⌊αdN⌉],T=∑i=1N⌊αdi⌉\mathrm{LR}(h_{1:N},d,\alpha)=\big[\underbrace{h_1,\dots,h_1}_{\lfloor\alpha d_1\rceil},\;\dots,\;\underbrace{h_N,\dots,h_N}_{\lfloor\alpha d_N\rceil}\big],\qquad T=\sum_{i=1}^{N}\lfloor\alpha d_i\rceil
hih_i
encoder state of phoneme ii
did_i
predicted duration in mel frames; FastSpeech 2 trains the duration predictor with MSE on log-durations
α\alpha
speed control: α>1\alpha>1 slows speech down, α<1\alpha<1 speeds it up
TT
number of mel frames; audio length is T⋅hopT\cdot\text{hop} samples

Vocoders. WaveNet generates one sample at a time with stacks of dilated causal convolutions (see Autoregressive generative models). It produces excellent audio but is very slow. HiFi-GAN upsamples the mel by the hop length (×256) with transposed convolutions and is trained as a Generative adversarial network (GAN). Its discriminators look at the waveform reshaped by periods 2, 3, 5, 7 and 11 (multi-period) and at several downsampled scales (multi-scale), plus a mel L1 loss. It runs far faster than real time on a GPU.

VITS trains one model from phonemes to waveform. It is a conditional VAE whose prior is enriched with a Normalizing flows stack and whose decoder is a HiFi-GAN-style generator trained adversarially. Monotonic alignment search finds durations during training, with no external aligner. A stochastic duration predictor lets the same text come out with different rhythms. VALL-E treats TTS as language modelling. Speech is encoded by the EnCodec neural codec into 8 codebooks of discrete tokens (see Vector-quantized VAE (VQ-VAE)). A decoder-only Transformer predicts the first codebook autoregressively from phonemes plus the tokens of a 3-second prompt of an unseen speaker, and a non-autoregressive model fills in the other 7. It was trained on 60,000 hours of LibriLight.

Data and evaluation

Benchmarks and metrics

LJSpeech
13,100 clips, about 24 h, one female English speaker, 22,050 Hz; the standard single-speaker benchmark
Typical features
80-band log-mel; LJSpeech recipes use n_fft 1024 and hop 256 (11.6 ms per frame)
MOS
mean opinion score: listeners rate naturalness 1 (bad) to 5 (excellent); report the mean with a 95% confidence interval, ±1.96 s/n\pm1.96\,s/\sqrt{n}
Also used
CMOS (side-by-side comparison), WER of an ASR model on the output (intelligibility), speaker-embedding cosine similarity (voice cloning)

Durations and mel filters, live

length regulator: “hello world” → 8 phonemes → 63 frames · α = 1.00HHd̂ 5 → 5AH0d̂ 4 → 4Ld̂ 6 → 6OW1d̂ 14 → 14Wd̂ 6 → 6ER1d̂ 12 → 12Ld̂ 7 → 7Dd̂ 9 → 9helloworld0.00 s0.25 s0.50 s0.75 s1.00 s1.25 sT = Σ round(α·d̂) = 63 frames × 256 = 16,128 samples = 0.731 s at 22.05 kHzone frame = 256 / 22,050 = 11.61 ms · exact α·Σd̂ = 63.00, rounding adds 0.00 frameseach phoneme’s hidden state is copied d times; the mel decoder then runs on all 63 frames in parallelmel filterbank · n_mels = 80 · n_fft = 1024 · 0–8 kHz at 22.05 kHzall filters, linear Hzzoom 0–400 Hz · ticks = FFT bins02k4k6k8k0 Hz200 Hz400 Hzfilter 1: centre 22.1 Hz, width 44.9 Hz · filter 80: centre 7733.5 Hz, width 524.8 Hz (11.7×)FFT bin spacing sr / n_fft = 21.53 Hz · narrowest filter covers 2 binsempty filters: 0 (every mel channel sees at least one FFT bin)equal steps in mel = 2595·log10(1 + f/700): dense below 1 kHz, sparse above, like pitch perception
apply α to
n_fft
fig.Top: the length regulator on "hello world" (8 phonemes). At α = 1 the predicted durations sum to 63 frames × 256 = 16,128 samples = 0.731 s. At α = 2 the same phonemes fill 126 frames (1.463 s). At α = 0.5 they fill 33 frames, not 31.5, because each duration is rounded separately (half up here). Applying α = 1.5 to "world" only gives 81 frames (0.940 s), which is how per-word emphasis is controlled. Bottom: the 80-filter mel bank with n_fft = 1024. Filter 1 is centred at 22.1 Hz and is 44.9 Hz wide, filter 80 at 7733.5 Hz is 524.8 Hz wide (11.7×), and the narrowest filter covers 2 FFT bins. With n_fft = 512 and 128 mels, 5 filters are empty.

Worked example

PhonemeHHAH0LOW1WER1LDTotal
predicted d546146127963
1.25 · d6.2557.517.57.5158.7511.2578.75
rounded6581881591180
Worked example: length regulator on "hello world" at α = 1.25 (22,050 Hz, hop 256)

So the decoder produces T=80T=80 mel frames, i.e. 80×256=20,48080\times256=20{,}480 samples =0.929=0.929 s, against 0.731 s at α=1\alpha=1. Rounding added 1.251.25 frames over the exact 78.7578.75. The stressed vowels OW1 and ER1 hold 33 of the 80 frames.

python
1import torch
2
3def length_regulator(h, d_hat, alpha=1.0):
4 """h: (N, d_model) phoneme states; d_hat: (N,) predicted durations in frames."""
5 d = torch.clamp(torch.round(d_hat * alpha), min=1).long()
6 return torch.repeat_interleave(h, d, dim=0), d # (sum(d), d_model)
7
8h = torch.randn(8, 256) # HH AH0 L OW1 W ER1 L D
9d_hat = torch.tensor([5., 4, 6, 14, 6, 12, 7, 9])
10frames, d = length_regulator(h, d_hat, alpha=1.25)
11print(d.tolist(), frames.shape) # [6, 5, 8, 18, 8, 15, 9, 11] torch.Size([80, 256])
12print(frames.shape[0] * 256 / 22050) # 0.9288 s of audio
A FastSpeech-style length regulator in PyTorch (not run here; the frame counts were checked separately)

Where it's used

Assistants and accessibility

Screen readers, voice assistants and navigation prompts, where FastSpeech-style parallel models keep latency low.

Voice cloning and dubbing

Zero-shot speaker adaptation from a few seconds of audio (VALL-E style prompts); this raises consent and spoofing concerns.

Data for ASR

Synthetic speech augments training data for Speech-to-text, especially for rare words and names.

Check yourself

A FastSpeech model predicts durations that sum to 200 frames for a sentence (hop 256, 22,050 Hz). With α = 0.8, about how long is the audio?

Check yourself

Why can Tacotron 2 occasionally skip or repeat words while FastSpeech 2 cannot?

Remember

  • ✓Pipeline: normalise text → G2P phonemes → acoustic model → 80-band mel → vocoder → waveform.
  • ✓Length regulator: repeat hih_i for ⌊αdi⌉\lfloor\alpha d_i\rceil frames; T=∑i⌊αdi⌉T=\sum_i\lfloor\alpha d_i\rceil frames, audio =T⋅hop=T\cdot\text{hop} samples.
  • ✓Tacotron 2 is autoregressive with attention (MOS 4.53 vs 4.58 human); FastSpeech 2 is parallel with duration, pitch and energy predictors.
  • ✓Vocoders: WaveNet (autoregressive, slow) → HiFi-GAN (GAN, multi-period discriminator with periods 2, 3, 5, 7, 11, faster than real time).
  • ✓VITS: one VAE + flow + GAN model; VALL-E: a language model over EnCodec tokens with a 3 s speaker prompt. Quality is judged by MOS with confidence intervals.
← back to Deep Learning roadmap