# Neural text-to-speech

> End-to-end and zero-shot TTS: VITS learns its own alignment (monotonic search) inside a VAE + flow + GAN, VALL-E predicts EnCodec tokens from a 3 s voice prompt, and flow models infill speech. Judge by MOS, ASR WER and speaker similarity.

Source: https://www.mlfoundry.in/learn/neural-text-to-speech  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 9 min read · updated 2026-10-15

**Neural text-to-speech (end-to-end and zero-shot):** covers TTS models that go beyond the two-stage acoustic-model-plus-vocoder pipeline. Three families dominate. **VITS** (Kim, Kong and Son, ICML 2021) maps phonemes straight to a waveform as one conditional VAE with a normalizing flow and adversarial training. **Codec language models** such as VALL-E (Wang et al., 2023, "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers") generate discrete audio-codec tokens. **Diffusion and flow-matching** models such as Grad-TTS (Popov et al., 2021) and Voicebox (Le et al., 2023) generate speech by iterative denoising. The last two can clone an unseen voice from a few seconds of audio.

The classic pipeline is covered in **Text-to-speech**: normalise text, convert to phonemes, predict a mel spectrogram (**Tacotron 2**, **FastSpeech 2**), then vocode it (**HiFi-GAN**). It has three weak points. The vocoder is trained on real mels but receives predicted ones at inference. Durations come from an external aligner or attention. And a new voice needs hours of that speaker's recordings, or at least fine-tuning.

End-to-end models remove the mismatch by training everything jointly on waveforms. Zero-shot models remove the need for per-speaker training: the voice is a conditioning input, given as a short audio prompt at inference time.

## VITS: one model, text to waveform

> Figure: Top: VITS in training. A posterior encoder maps the linear spectrogram (513 bins for n_fft 1024) to a latent z, which a HiFi-GAN generator decodes to audio against a multi-period discriminator. A flow maps z into the space of the text prior, and monotonic alignment search decides which phoneme's μ, σ each frame uses. At inference the posterior encoder is dropped: durations are sampled, the prior is expanded, sampled and inverted through the flow, then decoded. Bottom: VALL-E. Phonemes plus the 225 × 8 EnCodec tokens of a 3 s prompt condition an autoregressive model for codebook 1 and a non-autoregressive model for codebooks 2–8, and the EnCodec decoder returns 24 kHz audio.

VITS is trained as a conditional VAE. The posterior encoder sees the target speech; the prior sees only the text. To make the text prior expressive enough, a normalizing flow (affine coupling layers, see **Normalizing flows**) transforms the latent before it is compared with the prior. The decoder is a HiFi-GAN generator working on random latent slices, trained with HiFi-GAN's discriminators, feature-matching loss and a mel L1 reconstruction loss weighted by 45.

$$
\log p_\theta(x\mid c)\;\ge\;\mathbb E_{q_\phi(z\mid x)}\big[\log p_\theta(x\mid z)\big]-\mathrm{KL}\big(q_\phi(z\mid x)\,\big\Vert\,p_\theta(z\mid c)\big),\qquad p_\theta(z\mid c)=\mathcal N\big(f_\theta(z);\mu_\theta(c),\sigma_\theta(c)\big)\Big|\det\frac{\partial f_\theta(z)}{\partial z}\Big|
$$

- $x,\,c$: target speech and the conditioning text (phonemes); this is the ELBO of a **Variational autoencoder (VAE)**
- $q_\phi(z\mid x)$: posterior encoder, run on the linear spectrogram; used only in training
- $f_\theta$: invertible flow; at inference the sampled prior latent is mapped back with $f_\theta^{-1}$
- $\mu_\theta(c),\sigma_\theta(c)$: per-phoneme statistics from the text encoder, repeated along the alignment found by MAS

**Monotonic alignment search (MAS)** comes from Glow-TTS (Kim, Kim, Kong and Yoon, 2020). It finds the alignment $A$ that maximises the likelihood of the latent frames under the phoneme priors, subject to two rules: the alignment moves forward in the text, and it never skips a phoneme. Dynamic programming solves this exactly in $O(NT)$. The resulting durations also train a **stochastic duration predictor**, a flow-based model, so the same sentence can be sampled with different rhythms.

$$
Q_{i,j}=\max\big(Q_{i,j-1},\;Q_{i-1,j-1}\big)+\log\mathcal N\big(z_j;\mu_i,\sigma_i\big),\qquad d_i=\big|\{\,j:A(j)=i\,\}\big|
$$

- $Q_{i,j}$: best log-likelihood of aligning frames $1..j$ with frame $j$ on phoneme $i$; $Q_{1,1}=\log\mathcal N(z_1;\mu_1,\sigma_1)$
- $\max(\cdot)$: stay on phoneme $i$, or advance from phoneme $i-1$. No other moves are allowed
- $d_i$: duration of phoneme $i$ = number of frames assigned to it, read off the backtracked path from $(N,T)$

> Figure: MAS on "hello world" as 8 tokens (HH AH L OW W ER L D) and 36 latent frames drawn around each token's 2-D prior mean. The two L tokens share one mean. At the default noise σ = 0.5 (seed 1), MAS recovers all 36 frames and the true durations 3 4 5 7 3 6 4 4, with path log-likelihood −8.79. Per-frame argmax gets only 27/36, jumps backwards 6 times, and never uses token 7 (the second L), because it cannot tell two identical priors apart. At σ = 0.3 argmax still misses those 4 frames (32/36). At σ = 0.9 MAS returns 4 3 6 7 1 7 4 4 (32/36): its path scores −26.45, higher than the true alignment's −28.49, so it is still the exact optimum.

Worked example: MAS for 3 phonemes (A, B, C) and 5 frames. The cells are Q values computed from L rows A = [−0.1, −0.3, −2.0, −2.5, −3.0], B = [−1.5, −0.2, −0.4, −1.8, −2.2], C = [−3.0, −2.5, −1.0, −0.2, −0.1]

| Phoneme | j = 1 | j = 2 | j = 3 | j = 4 | j = 5 |
| --- | --- | --- | --- | --- | --- |
| A | −0.1 | −0.4 | −2.4 | −4.9 | −7.9 |
| B | −∞ | −0.3 | −0.7 | −2.5 | −4.7 |
| C | −∞ | −∞ | −1.3 | −0.9 | −1.0 |

For example, $Q_{C,4}=\max(Q_{C,3},Q_{B,3})+L_{C,4}=\max(-1.3,-0.7)-0.2=-0.9$. Backtracking from $Q_{C,5}=-1.0$ gives the path A, B, B, C, C, so the durations are $d=(1,2,2)$. The path's total is $-0.1-0.2-0.4-0.2-0.1=-1.0$, equal to $Q_{C,5}$. Cells with $i>j$ are $-\infty$ because no phoneme may be skipped.

## Codec language models: VALL-E

VALL-E treats TTS as conditional language modelling over the tokens of a **Neural codec models** encoder. EnCodec at 24 kHz and 6 kbps produces 75 frames per second, each quantised by 8 residual codebooks of 1024 entries. Codebook 1 carries most of the content and speaker identity; later codebooks add acoustic detail. The model was trained on the 60,000 hours of LibriLight audiobook speech, with transcripts produced by an ASR system.

$$
p(C\mid x,\tilde C)=\underbrace{\prod_{t=1}^{T}p\big(c_{t,1}\mid c_{<t,1},\,\tilde c_{:,1},\,x\big)}_{\text{AR, codebook 1}}\;\cdot\;\underbrace{\prod_{j=2}^{8}p\big(c_{:,j}\mid c_{:,<j},\,x,\,\tilde C\big)}_{\text{NAR, one pass per codebook}}
$$

- $x$: phoneme sequence of the text to speak
- $\tilde C$: codec tokens of the 3 s acoustic prompt: $225\times8$ tokens that fix the voice, room and microphone
- $C$: $T\times8$ tokens to generate; codebook 1 is sampled one frame at a time, codebooks 2–8 in 7 parallel passes

Because it samples like a language model, VALL-E produces varied prosody and copies the prompt's acoustic environment and emotion. It also inherits language-model failure modes: words can be repeated or skipped, and output changes from run to run. Later codec-LM systems add duration or alignment constraints for this reason.

## Diffusion and flow matching

**Grad-TTS** (Popov et al., 2021) keeps a Glow-TTS-style encoder and MAS, and replaces the decoder with a score-based diffusion model that turns noise into a mel conditioned on the aligned encoder output (see **Diffusion models**). **Voicebox** (Le et al., 2023) uses **Flow matching** to fill in a masked span of speech given the text and the surrounding audio. One model then handles zero-shot TTS (the prompt is the unmasked part), editing and denoising. NaturalSpeech 2 (Shen et al., 2023) runs latent diffusion over continuous codec latents. These models are non-autoregressive, so length is fixed in advance and they cannot loop.

**Reference numbers**

- **VITS data:** LJSpeech (single speaker, about 24 h) and VCTK (109 English speakers, about 44 h)
- **EnCodec in VALL-E:** 24 kHz, 75 frames/s, 8 codebooks × 1024 entries: $75\times8\times10$ bits $=6$ kbps
- **VALL-E:** 60k h of LibriLight, over 7,000 speakers; 3 s enrolment prompt; compared with YourTTS on LibriSpeech and VCTK
- **Voicebox:** flow matching; 60k h of English audiobooks, plus a multilingual model on 50k h in 6 languages
- **Release:** the VALL-E and Voicebox authors did not release their models, citing impersonation risk

## Evaluation

$$
\mathrm{SIM}=\cos\big(e(\hat y),\,e(y_{\text{ref}})\big)=\frac{e(\hat y)^\top e(y_{\text{ref}})}{\lVert e(\hat y)\rVert\,\lVert e(y_{\text{ref}})\rVert},\qquad \mathrm{WER}_{\text{TTS}}=\mathrm{WER}\big(\mathrm{ASR}(\hat y),\,x\big)
$$

- $e(\cdot)$: speaker embedding from a speaker-verification model (VALL-E used a WavLM-TDNN verifier); $\hat y$ is synthetic, $y_{\text{ref}}$ is real speech of the target speaker
- $\mathrm{ASR}$: a fixed recogniser (often a HuBERT-CTC model or Whisper); its errors on synthetic speech measure intelligibility. CER is used for languages without spaces

Subjective tests are still the reference. **MOS** asks listeners to rate naturalness from 1 to 5. **CMOS** asks them to compare two systems side by side on a −3 to +3 scale, which is more sensitive between strong systems. **SMOS** rates similarity to the target voice. Learned MOS predictors such as UTMOS (Saeki et al., 2022) are used for quick comparisons, not as a replacement.

```python
import numpy as np

def mas(L):
    """L: (N, T) log-likelihoods log N(z_j; mu_i, sigma_i). Returns durations (N,)."""
    N, T = L.shape
    Q = np.full((N, T), -np.inf)
    Q[0, 0] = L[0, 0]
    for j in range(1, T):
        for i in range(min(j + 1, N)):
            prev = Q[i, j - 1] if i == 0 else max(Q[i, j - 1], Q[i - 1, j - 1])
            Q[i, j] = prev + L[i, j]
    i, owner = N - 1, np.zeros(T, dtype=int)
    for j in range(T - 1, -1, -1):           # backtrack from the last frame and token
        owner[j] = i
        if i > 0 and (i == j or Q[i, j - 1] < Q[i - 1, j - 1]):
            i -= 1
    return np.bincount(owner, minlength=N), Q

L = np.array([[-0.1, -0.3, -2.0, -2.5, -3.0],
              [-1.5, -0.2, -0.4, -1.8, -2.2],
              [-3.0, -2.5, -1.0, -0.2, -0.1]])
d, Q = mas(L)
print(d)                                 # [1 2 2]
```

Monotonic alignment search in numpy (run here on the worked example: it prints durations [1 2 2] and the Q table above)

The three families side by side

| Family | Generates | Alignment | Voice |
| --- | --- | --- | --- |
| VITS (VAE + flow + GAN) | waveform in one parallel pass | MAS in training, sampled durations at inference | fixed speakers, or a speaker ID |
| Codec LM (VALL-E) | discrete codec tokens, AR then NAR | implicit, learned by the language model | zero-shot from a 3 s prompt |
| Diffusion / flow (Grad-TTS, Voicebox) | mel or latents by iterative denoising | MAS or a duration model; Voicebox infills a fixed-length span | zero-shot via audio context (Voicebox) |

> **Zero-shot scores are easy to misread:** SIM depends on which verifier computes the embeddings, WER depends on which ASR model transcribes, and both depend on the prompt length and test set. A model can score high SIM by copying the prompt's noise and channel. Compare systems only under the same verifier, ASR model, prompts and test utterances, and back the numbers with a listening test.

> **Voice cloning needs consent:** A few seconds of someone's voice is enough to make these systems speak as them. That enables fraud, impersonation and spoofing of voice-based authentication. Responsible deployment means consent from the voice owner, audible or inaudible watermarking of generated audio, and synthetic-speech detection of the kind benchmarked by the ASVspoof challenges.

**Question:** MAS forbids one transition that a per-frame argmax allows. Which one, and why does it matter for duplicated phonemes?

- A. Staying on the same phoneme; it forces every phoneme to last one frame
- B. Moving backwards or skipping ahead in the text; with two identical L priors, only the order constraint can decide which L a frame belongs to
- C. Moving to the next phoneme; it keeps the whole sentence on one phoneme
- D. None; MAS and argmax always agree

**Answer:** B. Moving backwards or skipping ahead in the text; with two identical L priors, only the order constraint can decide which L a frame belongs to — Each step either stays on phoneme $i$ or advances to $i+1$. Two phonemes with the same prior give identical likelihood rows, so argmax always picks the first. MAS uses position in the sequence to assign the later frames to the second L.

**Question:** VALL-E generates 5 s of speech with EnCodec at 75 frames/s and 8 codebooks. How many sequential steps does the autoregressive stage take?

- A. 40
- B. 375
- C. 3000
- D. 3000 + 225

**Answer:** B. 375 — Only codebook 1 is autoregressive: $5\times75=375$ steps. Codebooks 2–8 ($7\times375=2625$ tokens) come from 7 non-autoregressive passes. Generating all $3000$ tokens one by one would be 8× slower.

**Key points**

- VITS: conditional VAE + flow-enhanced text prior + HiFi-GAN decoder with adversarial training; one model, phonemes to waveform.
- MAS: $Q_{i,j}=\max(Q_{i,j-1},Q_{i-1,j-1})+\log\mathcal N(z_j;\mu_i,\sigma_i)$, monotonic and non-skipping, $O(NT)$; durations train a stochastic duration predictor.
- VALL-E: language model over EnCodec tokens (8 × 1024 codebooks, 75 Hz); AR for codebook 1, NAR for 2–8; a 3 s prompt sets the voice.
- Grad-TTS and Voicebox generate by diffusion or flow matching; Voicebox infills masked speech for TTS, editing and denoising.
- Evaluate with MOS/CMOS/SMOS, WER from a fixed ASR model, and speaker similarity as a cosine of verifier embeddings; cloning requires consent and safeguards.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
