Neural text-to-speech

End-to-end and zero-shot TTS: VITS learns its own alignment (monotonic search) inside a VAE + flow + GAN, VALL-E predicts EnCodec tokens from a 3 s voice prompt, and flow models infill speech. Judge by MOS, ASR WER and speaker similarity.

9 min read873 wordsadvanced0 viewsupdated 2026-10-15

Definition

Neural text-to-speech (end-to-end and zero-shot) covers TTS models that go beyond the two-stage acoustic-model-plus-vocoder pipeline. Three families dominate. VITS (Kim, Kong and Son, ICML 2021) maps phonemes straight to a waveform as one conditional VAE with a normalizing flow and adversarial training. Codec language models such as VALL-E (Wang et al., 2023, "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers") generate discrete audio-codec tokens. Diffusion and flow-matching models such as Grad-TTS (Popov et al., 2021) and Voicebox (Le et al., 2023) generate speech by iterative denoising. The last two can clone an unseen voice from a few seconds of audio.

The classic pipeline is covered in Text-to-speech: normalise text, convert to phonemes, predict a mel spectrogram (Tacotron 2, FastSpeech 2), then vocode it (HiFi-GAN). It has three weak points. The vocoder is trained on real mels but receives predicted ones at inference. Durations come from an external aligner or attention. And a new voice needs hours of that speaker's recordings, or at least fine-tuning.

End-to-end models remove the mismatch by training everything jointly on waveforms. Zero-shot models remove the need for per-speaker training: the voice is a conditioning input, given as a short audio prompt at inference time.

VITS: one model, text to waveform

VITS (Kim, Kong & Son, 2021): conditional VAE + normalizing flow + GAN in one modellinear spec513 × T (train only)posterior encoderq(z | x_lin)zdecoderHiFi-GAN generatorwaveformDMPDflow f (coupling)f(z): invertiblemonotonic alignmentsearch (MAS) → durationstext encoder (Transformer)phonemes → μ_i, σ_iprior p(z | c) = N(f(z); μ_A(j), σ_A(j)) · |det ∂f/∂z|stochastic duration predictorloss: mel L1 (×45) + KL + duration + adversarial + feature matchinginference: phonemes → μ, σ → sampled durations → expand → sample → f⁻¹ → decoder → audioVALL-E (Wang et al., 2023): language model over EnCodec tokens, 8 codebooks × 75 Hzphonemes xof the new text3 s promptunseen speakerEnCodec encoder6 kbps, 24 kHzprompt tokens C̃ (225 × 8)generated tokens C (T × 8)181024-entry codebooks · 13.3 ms/colAR Transformercodebook 1, one by oneNAR Transformercodebook j = 2…8, 7 passesEnCodec decoder→ 24 kHz waveform60k h of LibriLight; the prompt sets voice, room and channel; no fine-tuning for the new speaker
fig.Top: VITS in training. A posterior encoder maps the linear spectrogram (513 bins for n_fft 1024) to a latent z, which a HiFi-GAN generator decodes to audio against a multi-period discriminator. A flow maps z into the space of the text prior, and monotonic alignment search decides which phoneme's μ, σ each frame uses. At inference the posterior encoder is dropped: durations are sampled, the prior is expanded, sampled and inverted through the flow, then decoded. Bottom: VALL-E. Phonemes plus the 225 × 8 EnCodec tokens of a 3 s prompt condition an autoregressive model for codebook 1 and a non-autoregressive model for codebooks 2–8, and the EnCodec decoder returns 24 kHz audio.

VITS is trained as a conditional VAE. The posterior encoder sees the target speech; the prior sees only the text. To make the text prior expressive enough, a normalizing flow (affine coupling layers, see Normalizing flows) transforms the latent before it is compared with the prior. The decoder is a HiFi-GAN generator working on random latent slices, trained with HiFi-GAN's discriminators, feature-matching loss and a mel L1 reconstruction loss weighted by 45.

log⁡pθ(x∣c)  ≥  Eqϕ(z∣x)[log⁡pθ(x∣z)]−KL(qϕ(z∣x) ∥ pθ(z∣c)),pθ(z∣c)=N(fθ(z);μθ(c),σθ(c))∣det⁡∂fθ(z)∂z∣\log p_\theta(x\mid c)\;\ge\;\mathbb E_{q_\phi(z\mid x)}\big[\log p_\theta(x\mid z)\big]-\mathrm{KL}\big(q_\phi(z\mid x)\,\big\Vert\,p_\theta(z\mid c)\big),\qquad p_\theta(z\mid c)=\mathcal N\big(f_\theta(z);\mu_\theta(c),\sigma_\theta(c)\big)\Big|\det\frac{\partial f_\theta(z)}{\partial z}\Big|
x, cx,\,c
target speech and the conditioning text (phonemes); this is the ELBO of a Variational autoencoder (VAE)
qϕ(z∣x)q_\phi(z\mid x)
posterior encoder, run on the linear spectrogram; used only in training
fθf_\theta
invertible flow; at inference the sampled prior latent is mapped back with fθ−1f_\theta^{-1}
μθ(c),σθ(c)\mu_\theta(c),\sigma_\theta(c)
per-phoneme statistics from the text encoder, repeated along the alignment found by MAS

Monotonic alignment search (MAS) comes from Glow-TTS (Kim, Kim, Kong and Yoon, 2020). It finds the alignment AA that maximises the likelihood of the latent frames under the phoneme priors, subject to two rules: the alignment moves forward in the text, and it never skips a phoneme. Dynamic programming solves this exactly in O(NT)O(NT). The resulting durations also train a stochastic duration predictor, a flow-based model, so the same sentence can be sampled with different rhythms.

Qi,j=max⁡(Qi,j−1,  Qi−1,j−1)+log⁡N(zj;μi,σi),di=∣{ j:A(j)=i }∣Q_{i,j}=\max\big(Q_{i,j-1},\;Q_{i-1,j-1}\big)+\log\mathcal N\big(z_j;\mu_i,\sigma_i\big),\qquad d_i=\big|\{\,j:A(j)=i\,\}\big|
Qi,jQ_{i,j}
best log-likelihood of aligning frames 1..j1..j with frame jj on phoneme ii; Q1,1=log⁡N(z1;μ1,σ1)Q_{1,1}=\log\mathcal N(z_1;\mu_1,\sigma_1)
max⁡(⋅)\max(\cdot)
stay on phoneme ii, or advance from phoneme i−1i-1. No other moves are allowed
did_i
duration of phoneme ii = number of frames assigned to it, read off the backtracked path from (N,T)(N,T)
log N(z_j; μ_i, I): token i (rows) × latent frame j (columns)outlined: MAS path · dots: per-frame argmax · noise σ = 0.5 · seed 11 HH2 AH3 L4 OW5 W6 ER7 L8 Dframe → token (red border: differs from the true alignment)true111222233333444444455566666677778888MAS111222233333444444455566666677778888argmax116226233333444344455566362633338888latent plane (2-D)HHAHLOWWERDMAS durations: 3 4 5 7 3 6 4 4 (true 3 4 5 7 3 6 4 4)MAS frames correct: 36/36 · path log-lik -8.79 vs true -8.79argmax correct: 27/36 · 6 backward jumps · 1 token never usedMAS recovers every duration. Argmax cannot tell the two L's apart:both have the same prior, so it labels token 7's frames as token 3.DP cost O(N·T) = 8·36 = 288 cells; durations d_i = frames on row i.In VITS the durations then train the duration predictor.
matrix
seed
fig.MAS on "hello world" as 8 tokens (HH AH L OW W ER L D) and 36 latent frames drawn around each token's 2-D prior mean. The two L tokens share one mean. At the default noise σ = 0.5 (seed 1), MAS recovers all 36 frames and the true durations 3 4 5 7 3 6 4 4, with path log-likelihood −8.79. Per-frame argmax gets only 27/36, jumps backwards 6 times, and never uses token 7 (the second L), because it cannot tell two identical priors apart. At σ = 0.3 argmax still misses those 4 frames (32/36). At σ = 0.9 MAS returns 4 3 6 7 1 7 4 4 (32/36): its path scores −26.45, higher than the true alignment's −28.49, so it is still the exact optimum.
Phonemej = 1j = 2j = 3j = 4j = 5
A−0.1−0.4−2.4−4.9−7.9
B−∞−0.3−0.7−2.5−4.7
C−∞−∞−1.3−0.9−1.0
Worked example: MAS for 3 phonemes (A, B, C) and 5 frames. The cells are Q values computed from L rows A = [−0.1, −0.3, −2.0, −2.5, −3.0], B = [−1.5, −0.2, −0.4, −1.8, −2.2], C = [−3.0, −2.5, −1.0, −0.2, −0.1]

For example, QC,4=max⁡(QC,3,QB,3)+LC,4=max⁡(−1.3,−0.7)−0.2=−0.9Q_{C,4}=\max(Q_{C,3},Q_{B,3})+L_{C,4}=\max(-1.3,-0.7)-0.2=-0.9. Backtracking from QC,5=−1.0Q_{C,5}=-1.0 gives the path A, B, B, C, C, so the durations are d=(1,2,2)d=(1,2,2). The path's total is −0.1−0.2−0.4−0.2−0.1=−1.0-0.1-0.2-0.4-0.2-0.1=-1.0, equal to QC,5Q_{C,5}. Cells with i>ji>j are −∞-\infty because no phoneme may be skipped.

Codec language models: VALL-E

VALL-E treats TTS as conditional language modelling over the tokens of a Neural codec models encoder. EnCodec at 24 kHz and 6 kbps produces 75 frames per second, each quantised by 8 residual codebooks of 1024 entries. Codebook 1 carries most of the content and speaker identity; later codebooks add acoustic detail. The model was trained on the 60,000 hours of LibriLight audiobook speech, with transcripts produced by an ASR system.

p(C∣x,C~)=∏t=1Tp(ct,1∣c<t,1, c~:,1, x)⏟AR, codebook 1  ⋅  ∏j=28p(c:,j∣c:,<j, x, C~)⏟NAR, one pass per codebookp(C\mid x,\tilde C)=\underbrace{\prod_{t=1}^{T}p\big(c_{t,1}\mid c_{<t,1},\,\tilde c_{:,1},\,x\big)}_{\text{AR, codebook 1}}\;\cdot\;\underbrace{\prod_{j=2}^{8}p\big(c_{:,j}\mid c_{:,<j},\,x,\,\tilde C\big)}_{\text{NAR, one pass per codebook}}
xx
phoneme sequence of the text to speak
C~\tilde C
codec tokens of the 3 s acoustic prompt: 225×8225\times8 tokens that fix the voice, room and microphone
CC
T×8T\times8 tokens to generate; codebook 1 is sampled one frame at a time, codebooks 2–8 in 7 parallel passes

Because it samples like a language model, VALL-E produces varied prosody and copies the prompt's acoustic environment and emotion. It also inherits language-model failure modes: words can be repeated or skipped, and output changes from run to run. Later codec-LM systems add duration or alignment constraints for this reason.

Diffusion and flow matching

Grad-TTS (Popov et al., 2021) keeps a Glow-TTS-style encoder and MAS, and replaces the decoder with a score-based diffusion model that turns noise into a mel conditioned on the aligned encoder output (see Diffusion models). Voicebox (Le et al., 2023) uses Flow matching to fill in a masked span of speech given the text and the surrounding audio. One model then handles zero-shot TTS (the prompt is the unmasked part), editing and denoising. NaturalSpeech 2 (Shen et al., 2023) runs latent diffusion over continuous codec latents. These models are non-autoregressive, so length is fixed in advance and they cannot loop.

Reference numbers

VITS data
LJSpeech (single speaker, about 24 h) and VCTK (109 English speakers, about 44 h)
EnCodec in VALL-E
24 kHz, 75 frames/s, 8 codebooks × 1024 entries: 75×8×1075\times8\times10 bits =6=6 kbps
VALL-E
60k h of LibriLight, over 7,000 speakers; 3 s enrolment prompt; compared with YourTTS on LibriSpeech and VCTK
Voicebox
flow matching; 60k h of English audiobooks, plus a multilingual model on 50k h in 6 languages
Release
the VALL-E and Voicebox authors did not release their models, citing impersonation risk

Evaluation

SIM=cos⁡(e(y^), e(yref))=e(y^)⊤e(yref)∥e(y^)∥ ∥e(yref)∥,WERTTS=WER(ASR(y^), x)\mathrm{SIM}=\cos\big(e(\hat y),\,e(y_{\text{ref}})\big)=\frac{e(\hat y)^\top e(y_{\text{ref}})}{\lVert e(\hat y)\rVert\,\lVert e(y_{\text{ref}})\rVert},\qquad \mathrm{WER}_{\text{TTS}}=\mathrm{WER}\big(\mathrm{ASR}(\hat y),\,x\big)
e(⋅)e(\cdot)
speaker embedding from a speaker-verification model (VALL-E used a WavLM-TDNN verifier); y^\hat y is synthetic, yrefy_{\text{ref}} is real speech of the target speaker
ASR\mathrm{ASR}
a fixed recogniser (often a HuBERT-CTC model or Whisper); its errors on synthetic speech measure intelligibility. CER is used for languages without spaces

Subjective tests are still the reference. MOS asks listeners to rate naturalness from 1 to 5. CMOS asks them to compare two systems side by side on a −3 to +3 scale, which is more sensitive between strong systems. SMOS rates similarity to the target voice. Learned MOS predictors such as UTMOS (Saeki et al., 2022) are used for quick comparisons, not as a replacement.

python
1import numpy as np
2
3def mas(L):
4 """L: (N, T) log-likelihoods log N(z_j; mu_i, sigma_i). Returns durations (N,)."""
5 N, T = L.shape
6 Q = np.full((N, T), -np.inf)
7 Q[0, 0] = L[0, 0]
8 for j in range(1, T):
9 for i in range(min(j + 1, N)):
10 prev = Q[i, j - 1] if i == 0 else max(Q[i, j - 1], Q[i - 1, j - 1])
11 Q[i, j] = prev + L[i, j]
12 i, owner = N - 1, np.zeros(T, dtype=int)
13 for j in range(T - 1, -1, -1): # backtrack from the last frame and token
14 owner[j] = i
15 if i > 0 and (i == j or Q[i, j - 1] < Q[i - 1, j - 1]):
16 i -= 1
17 return np.bincount(owner, minlength=N), Q
18
19L = np.array([[-0.1, -0.3, -2.0, -2.5, -3.0],
20 [-1.5, -0.2, -0.4, -1.8, -2.2],
21 [-3.0, -2.5, -1.0, -0.2, -0.1]])
22d, Q = mas(L)
23print(d) # [1 2 2]
Monotonic alignment search in numpy (run here on the worked example: it prints durations [1 2 2] and the Q table above)
FamilyGeneratesAlignmentVoice
VITS (VAE + flow + GAN)waveform in one parallel passMAS in training, sampled durations at inferencefixed speakers, or a speaker ID
Codec LM (VALL-E)discrete codec tokens, AR then NARimplicit, learned by the language modelzero-shot from a 3 s prompt
Diffusion / flow (Grad-TTS, Voicebox)mel or latents by iterative denoisingMAS or a duration model; Voicebox infills a fixed-length spanzero-shot via audio context (Voicebox)
The three families side by side

Check yourself

MAS forbids one transition that a per-frame argmax allows. Which one, and why does it matter for duplicated phonemes?

Check yourself

VALL-E generates 5 s of speech with EnCodec at 75 frames/s and 8 codebooks. How many sequential steps does the autoregressive stage take?

Remember

  • ✓VITS: conditional VAE + flow-enhanced text prior + HiFi-GAN decoder with adversarial training; one model, phonemes to waveform.
  • ✓MAS: Qi,j=max⁡(Qi,j−1,Qi−1,j−1)+log⁡N(zj;μi,σi)Q_{i,j}=\max(Q_{i,j-1},Q_{i-1,j-1})+\log\mathcal N(z_j;\mu_i,\sigma_i), monotonic and non-skipping, O(NT)O(NT); durations train a stochastic duration predictor.
  • ✓VALL-E: language model over EnCodec tokens (8 × 1024 codebooks, 75 Hz); AR for codebook 1, NAR for 2–8; a 3 s prompt sets the voice.
  • ✓Grad-TTS and Voicebox generate by diffusion or flow matching; Voicebox infills masked speech for TTS, editing and denoising.
  • ✓Evaluate with MOS/CMOS/SMOS, WER from a fixed ASR model, and speaker similarity as a cosine of verifier embeddings; cloning requires consent and safeguards.
← back to Deep Learning roadmap