Definition
Neural text-to-speech (end-to-end and zero-shot) covers TTS models that go beyond the two-stage acoustic-model-plus-vocoder pipeline. Three families dominate. VITS (Kim, Kong and Son, ICML 2021) maps phonemes straight to a waveform as one conditional VAE with a normalizing flow and adversarial training. Codec language models such as VALL-E (Wang et al., 2023, "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers") generate discrete audio-codec tokens. Diffusion and flow-matching models such as Grad-TTS (Popov et al., 2021) and Voicebox (Le et al., 2023) generate speech by iterative denoising. The last two can clone an unseen voice from a few seconds of audio.
The classic pipeline is covered in Text-to-speech: normalise text, convert to phonemes, predict a mel spectrogram (Tacotron 2, FastSpeech 2), then vocode it (HiFi-GAN). It has three weak points. The vocoder is trained on real mels but receives predicted ones at inference. Durations come from an external aligner or attention. And a new voice needs hours of that speaker's recordings, or at least fine-tuning.
End-to-end models remove the mismatch by training everything jointly on waveforms. Zero-shot models remove the need for per-speaker training: the voice is a conditioning input, given as a short audio prompt at inference time.
VITS: one model, text to waveform
VITS is trained as a conditional VAE. The posterior encoder sees the target speech; the prior sees only the text. To make the text prior expressive enough, a normalizing flow (affine coupling layers, see Normalizing flows) transforms the latent before it is compared with the prior. The decoder is a HiFi-GAN generator working on random latent slices, trained with HiFi-GAN's discriminators, feature-matching loss and a mel L1 reconstruction loss weighted by 45.
- target speech and the conditioning text (phonemes); this is the ELBO of a Variational autoencoder (VAE)
- posterior encoder, run on the linear spectrogram; used only in training
- invertible flow; at inference the sampled prior latent is mapped back with
- per-phoneme statistics from the text encoder, repeated along the alignment found by MAS
Monotonic alignment search (MAS) comes from Glow-TTS (Kim, Kim, Kong and Yoon, 2020). It finds the alignment that maximises the likelihood of the latent frames under the phoneme priors, subject to two rules: the alignment moves forward in the text, and it never skips a phoneme. Dynamic programming solves this exactly in . The resulting durations also train a stochastic duration predictor, a flow-based model, so the same sentence can be sampled with different rhythms.
- best log-likelihood of aligning frames with frame on phoneme ;
- stay on phoneme , or advance from phoneme . No other moves are allowed
- duration of phoneme = number of frames assigned to it, read off the backtracked path from
| Phoneme | j = 1 | j = 2 | j = 3 | j = 4 | j = 5 |
|---|---|---|---|---|---|
| A | −0.1 | −0.4 | −2.4 | −4.9 | −7.9 |
| B | −∞ | −0.3 | −0.7 | −2.5 | −4.7 |
| C | −∞ | −∞ | −1.3 | −0.9 | −1.0 |
For example, . Backtracking from gives the path A, B, B, C, C, so the durations are . The path's total is , equal to . Cells with are because no phoneme may be skipped.
Codec language models: VALL-E
VALL-E treats TTS as conditional language modelling over the tokens of a Neural codec models encoder. EnCodec at 24 kHz and 6 kbps produces 75 frames per second, each quantised by 8 residual codebooks of 1024 entries. Codebook 1 carries most of the content and speaker identity; later codebooks add acoustic detail. The model was trained on the 60,000 hours of LibriLight audiobook speech, with transcripts produced by an ASR system.
- phoneme sequence of the text to speak
- codec tokens of the 3 s acoustic prompt: tokens that fix the voice, room and microphone
- tokens to generate; codebook 1 is sampled one frame at a time, codebooks 2–8 in 7 parallel passes
Because it samples like a language model, VALL-E produces varied prosody and copies the prompt's acoustic environment and emotion. It also inherits language-model failure modes: words can be repeated or skipped, and output changes from run to run. Later codec-LM systems add duration or alignment constraints for this reason.
Diffusion and flow matching
Grad-TTS (Popov et al., 2021) keeps a Glow-TTS-style encoder and MAS, and replaces the decoder with a score-based diffusion model that turns noise into a mel conditioned on the aligned encoder output (see Diffusion models). Voicebox (Le et al., 2023) uses Flow matching to fill in a masked span of speech given the text and the surrounding audio. One model then handles zero-shot TTS (the prompt is the unmasked part), editing and denoising. NaturalSpeech 2 (Shen et al., 2023) runs latent diffusion over continuous codec latents. These models are non-autoregressive, so length is fixed in advance and they cannot loop.
Reference numbers
- VITS data
- LJSpeech (single speaker, about 24 h) and VCTK (109 English speakers, about 44 h)
- EnCodec in VALL-E
- 24 kHz, 75 frames/s, 8 codebooks × 1024 entries: bits kbps
- VALL-E
- 60k h of LibriLight, over 7,000 speakers; 3 s enrolment prompt; compared with YourTTS on LibriSpeech and VCTK
- Voicebox
- flow matching; 60k h of English audiobooks, plus a multilingual model on 50k h in 6 languages
- Release
- the VALL-E and Voicebox authors did not release their models, citing impersonation risk
Evaluation
- speaker embedding from a speaker-verification model (VALL-E used a WavLM-TDNN verifier); is synthetic, is real speech of the target speaker
- a fixed recogniser (often a HuBERT-CTC model or Whisper); its errors on synthetic speech measure intelligibility. CER is used for languages without spaces
Subjective tests are still the reference. MOS asks listeners to rate naturalness from 1 to 5. CMOS asks them to compare two systems side by side on a −3 to +3 scale, which is more sensitive between strong systems. SMOS rates similarity to the target voice. Learned MOS predictors such as UTMOS (Saeki et al., 2022) are used for quick comparisons, not as a replacement.
1import numpy as np23def mas(L):4 """L: (N, T) log-likelihoods log N(z_j; mu_i, sigma_i). Returns durations (N,)."""5 N, T = L.shape6 Q = np.full((N, T), -np.inf)7 Q[0, 0] = L[0, 0]8 for j in range(1, T):9 for i in range(min(j + 1, N)):10 prev = Q[i, j - 1] if i == 0 else max(Q[i, j - 1], Q[i - 1, j - 1])11 Q[i, j] = prev + L[i, j]12 i, owner = N - 1, np.zeros(T, dtype=int)13 for j in range(T - 1, -1, -1): # backtrack from the last frame and token14 owner[j] = i15 if i > 0 and (i == j or Q[i, j - 1] < Q[i - 1, j - 1]):16 i -= 117 return np.bincount(owner, minlength=N), Q1819L = np.array([[-0.1, -0.3, -2.0, -2.5, -3.0],20 [-1.5, -0.2, -0.4, -1.8, -2.2],21 [-3.0, -2.5, -1.0, -0.2, -0.1]])22d, Q = mas(L)23print(d) # [1 2 2]| Family | Generates | Alignment | Voice |
|---|---|---|---|
| VITS (VAE + flow + GAN) | waveform in one parallel pass | MAS in training, sampled durations at inference | fixed speakers, or a speaker ID |
| Codec LM (VALL-E) | discrete codec tokens, AR then NAR | implicit, learned by the language model | zero-shot from a 3 s prompt |
| Diffusion / flow (Grad-TTS, Voicebox) | mel or latents by iterative denoising | MAS or a duration model; Voicebox infills a fixed-length span | zero-shot via audio context (Voicebox) |
Check yourself
MAS forbids one transition that a per-frame argmax allows. Which one, and why does it matter for duplicated phonemes?
Check yourself
VALL-E generates 5 s of speech with EnCodec at 75 frames/s and 8 codebooks. How many sequential steps does the autoregressive stage take?
Remember
- ✓VITS: conditional VAE + flow-enhanced text prior + HiFi-GAN decoder with adversarial training; one model, phonemes to waveform.
- ✓MAS: , monotonic and non-skipping, ; durations train a stochastic duration predictor.
- ✓VALL-E: language model over EnCodec tokens (8 × 1024 codebooks, 75 Hz); AR for codebook 1, NAR for 2–8; a 3 s prompt sets the voice.
- ✓Grad-TTS and Voicebox generate by diffusion or flow matching; Voicebox infills masked speech for TTS, editing and denoising.
- ✓Evaluate with MOS/CMOS/SMOS, WER from a fixed ASR model, and speaker similarity as a cosine of verifier embeddings; cloning requires consent and safeguards.