# Audio generation

> General audio generators either model codec tokens autoregressively (AudioLM, AudioGen) or run latent diffusion over mel latents (AudioLDM), steered by T5 or CLAP text embeddings and classifier-free guidance, and are scored by FAD, KL and CLAP score.

Source: https://www.mlfoundry.in/learn/audio-generation  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 8 min read · updated 2026-10-15

**Audio generation:** is learning a distribution over waveforms, $p(x)$ or $p(x\mid c)$ for a condition $c$ such as a text prompt, and sampling sounds from it: speech continuations, sound effects, ambiences, music. Two families dominate. **Token language models** generate the discrete codes of a neural codec, as in AudioLM (Borsos et al., 2022) and AudioGen (Kreuk et al., ICLR 2023). **Latent diffusion models** denoise a compressed spectrogram representation, as in AudioLDM (Liu et al., ICML 2023).

Raw audio is a hard target. One second at 16 kHz is 16,000 samples, and a sound has structure at every scale: sample-level periodicity, phonemes or note onsets every tens of milliseconds, and words, rhythm or scenes over seconds. **WaveNet** models samples directly. Sampling is slow, and without conditioning it produces speech-like babble with no long-range structure. Modern systems first compress audio into a much shorter representation, either the discrete tokens of a **Neural codec models** codec or the continuous latent of a VAE over a **Mel spectrograms** image, and generate in that space.

## Tokens: AudioLM and AudioGen

AudioLM uses two tokenizers. **Semantic tokens** come from k-means (1,024 clusters) on intermediate features of a self-supervised w2v-BERT model, at 25 per second. They capture content and long-range structure but cannot be decoded to good audio. **Acoustic tokens** are the RVQ codes of SoundStream, which carry speaker, timbre and recording conditions and decode to a waveform. Generation is split into three autoregressive Transformer stages:

$$
p(x)\;\approx\;\underbrace{p(s_{1:T_s})}_{\text{semantic}}\;\underbrace{p\big(a^{1:4}_{1:T_a}\mid s\big)}_{\text{coarse acoustic}}\;\underbrace{p\big(a^{5:12}_{1:T_a}\mid a^{1:4}\big)}_{\text{fine acoustic}}
$$

- $s_{1:T_s}$: semantic tokens, 25 per second
- $a^{q}_{t}$: SoundStream code of RVQ level $q$ at frame $t$ (50 frames per second, 12 levels of 1,024 entries)
- $\approx$: the fine stage ignores the semantic tokens; each factor is one decoder-only Transformer

> Figure: The two families. A: AudioLM generates semantic tokens, then the first 4 SoundStream RVQ levels, then levels 5–12, and the codec decodes the result; AudioGen replaces the semantic stage with a text encoder. B: AudioLDM denoises a VAE latent of a mel spectrogram under a CLAP embedding, then a VAE decoder and HiFi-GAN produce the waveform. Bottom: classifier-free guidance and the three standard metrics.

Prompted with a few seconds of speech, AudioLM continues it with the same voice and recording conditions and with syntactically plausible content, without any transcripts in training. AudioGen conditions on text instead. A T5 encoder embeds the description, the Transformer cross-attends to it, and generation runs over the tokens of a 16 kHz codec. AudioGen also mixes pairs of training clips and joins their captions, so the model learns to compose sources ("a dog barks while rain falls").

## Latent diffusion: AudioLDM

AudioLDM treats the mel spectrogram as an image. A VAE compresses it to a latent $z_0$, and a U-Net is trained to remove noise from $z_t$ as in **Latent diffusion models**. A **HiFi-GAN** vocoder turns the decoded mel into 16 kHz audio. The condition is a CLAP embedding. CLAP is trained contrastively, like **CLIP**, so that text and audio embeddings of matching pairs are close. AudioLDM trains with the CLAP embedding of the audio itself and swaps in the embedding of the text prompt at sampling time, so training needs no captions.

$$
\mathcal L=\mathbb E_{z_0,\epsilon,t}\big\|\epsilon-\epsilon_\theta(z_t,t,E)\big\|^2,\qquad \tilde\epsilon=\epsilon_\theta(z_t,t,\varnothing)+w\big(\epsilon_\theta(z_t,t,E)-\epsilon_\theta(z_t,t,\varnothing)\big)
$$

- $z_t$: noised latent of the mel spectrogram at diffusion step $t$
- $E$: CLAP embedding: of the audio in training, of the text prompt at sampling time
- $\varnothing$: the condition dropped; training drops it at random so one network learns both terms
- $w$: guidance scale. $w=1$ is plain conditional sampling; $w>1$ follows the prompt more closely at the cost of variety. Token models apply the same rule to logits

Later systems keep the recipe and change the parts: larger text encoders, transformer denoisers, or latents of a waveform autoencoder instead of mels. Stable Audio, for example, adds timing conditioning so that the length of the output can be chosen.

**The reference systems**

- **AudioLM:** w2v-BERT semantic tokens (k-means, 1,024) at 25 Hz; SoundStream acoustic tokens, 12 RVQ levels at 50 Hz; three Transformer stages (semantic, coarse levels 1–4, fine levels 5–12); speech trained on LibriLight without transcripts
- **AudioGen:** text → T5 encoder → autoregressive Transformer over 16 kHz codec tokens; classifier-free guidance on logits; caption-mixing augmentation
- **AudioLDM:** VAE over mel spectrograms, latent diffusion U-Net conditioned on CLAP, HiFi-GAN vocoder, 16 kHz; evaluated on AudioCaps
- **FAD:** Kilgour et al., Interspeech 2019: Fréchet distance between Gaussians fitted to 128-d VGGish embeddings of real and generated sets
- **KL and CLAP score:** KL between a pretrained tagger's label distributions for reference and generated clips of the same prompt; CLAP score: cosine between the prompt's and the clip's CLAP embeddings

## Evaluation

There is no single target waveform for "rain on a tin roof", so metrics compare distributions or check adherence. **FAD** embeds every real and every generated clip, fits a Gaussian to each set and computes the Fréchet distance between them. Lower is better:

$$
\mathrm{FAD}=\big\|\mu_r-\mu_g\big\|^2+\operatorname{tr}\!\Big(\Sigma_r+\Sigma_g-2\big(\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2}\big)^{1/2}\Big)
$$

- $\mu_r,\Sigma_r$: mean and covariance of the real clips' embeddings
- $\mu_g,\Sigma_g$: the same for generated clips
- $\operatorname{tr}(\cdot)$: zero when the covariances are equal; grows when the generator has too little or too much variety

FAD needs no pairing, but it says nothing about whether a clip matches its prompt. **KL** runs a pretrained audio tagger (for example PANNs or PaSST) on the reference and the generated clip for the same prompt and compares the label distributions. **CLAP score** measures text–audio agreement directly. Listening tests with overall-quality and text-relevance ratings remain the final check.

> Figure: FAD between toy 4-D embeddings: real clips come from two classes, the generator is the same mixture with a shift, a spread factor and a class share. Default (shift 0.5, 200 clips per set): FAD 0.355 from samples against 0.250 in closed form, all of it in the mean term; two real sets of 200 already give 0.086. With 50 clips an exact copy of the real distribution scores 0.814, and with 1,000 it scores 0.017. Spread ×0.5 moves only the covariance term (closed form 0.505); generating only class A gives 2.142 in closed form, split 1.480 / 0.662.

## Worked example: FAD by hand

1. Real embeddings: $\mu_r=(0,0)$, $\Sigma_r=\mathrm{diag}(1,1)$. Generated: $\mu_g=(0.5,0)$, $\Sigma_g=\mathrm{diag}(0.25,1)$, so the generator is shifted and too narrow in the first dimension.
2. Mean term: $\|\mu_r-\mu_g\|^2=0.5^2+0^2=0.25$.
3. Diagonal covariances commute, so $(\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2})^{1/2}=\mathrm{diag}(\sqrt{0.25},\sqrt1)=\mathrm{diag}(0.5,1)$.
4. Covariance term: $\operatorname{tr}\Sigma_r+\operatorname{tr}\Sigma_g-2\cdot1.5=2+1.25-3=0.25$.
5. $\mathrm{FAD}=0.25+0.25=0.5$. The two failures cost the same here; per dimension the covariance term is $(\sigma_r-\sigma_g)^2=(1-0.5)^2$.

```python
import numpy as np

def frechet_audio_distance(emb_real, emb_gen):
    """FAD between two sets of clip embeddings, shape (n_clips, dim), e.g. 128-d VGGish."""
    mu_r, mu_g = emb_real.mean(0), emb_gen.mean(0)
    S_r, S_g = np.cov(emb_real, rowvar=False), np.cov(emb_gen, rowvar=False)
    w, V = np.linalg.eigh(S_r)
    h = (V * np.sqrt(np.clip(w, 0, None))) @ V.T          # S_r^(1/2)
    cross = np.sqrt(np.clip(np.linalg.eigvalsh(h @ S_g @ h), 0, None)).sum()
    return float(((mu_r - mu_g) ** 2).sum() + np.trace(S_r) + np.trace(S_g) - 2 * cross)

rng = np.random.default_rng(0)
real = rng.normal(size=(2000, 128))
print(round(frechet_audio_distance(real, rng.normal(size=(2000, 128))), 2))        # 4.24: sampling floor
print(round(frechet_audio_distance(real, rng.normal(size=(2000, 128)) + 0.2), 2))  # 9.35: shifted
print(round(frechet_audio_distance(real, 0.7 * rng.normal(size=(2000, 128))), 2))  # 14.52: too little variety

# Sampling a text-to-audio model (diffusers, not run here: needs torch and a download):
# from diffusers import AudioLDMPipeline
# pipe = AudioLDMPipeline.from_pretrained("cvssp/audioldm-s-full-v2")
# audio = pipe("rain on a tin roof", num_inference_steps=10, audio_length_in_s=5.0).audios[0]  # 16 kHz
```

FAD from two embedding matrices (numpy, runs as is; in practice the rows are VGGish or CLAP embeddings of real and generated clips)

> **FAD numbers are only comparable under one protocol:** FAD depends on the embedding model (VGGish, PANNs, CLAP), the sample rate and length of the clips, and the number of clips. With few clips the estimate is biased upwards: in the code above two sets drawn from the same distribution score 4.24, not 0. Compare systems only with the same embedding, the same reference set and similar set sizes.

**Strengths**

- Token LMs: one Transformer recipe for speech, sounds and music; prompting with audio continues voice and acoustics; streaming is natural.
- Latent diffusion: parallel generation of the whole clip, fewer sequential steps, easy inpainting and editing in the latent.

**Limitations**

- Token LMs: thousands of tokens per second of audio across codebooks; sampling is sequential and errors can compound.
- Latent diffusion: fixed-length windows, quality capped by the VAE and vocoder; many denoising steps unless distilled.

**Question:** Why does AudioLM use semantic tokens in addition to SoundStream tokens?

- A. SoundStream tokens cannot be decoded to audio
- B. Semantic tokens are needed for the codec's adversarial loss
- C. Modelling acoustic tokens alone keeps good audio quality but loses long-term coherence; coarse semantic tokens carry the structure, acoustic tokens the detail
- D. Semantic tokens double the bitrate

**Answer:** C. Modelling acoustic tokens alone keeps good audio quality but loses long-term coherence; coarse semantic tokens carry the structure, acoustic tokens the detail — Acoustic codes are many per second and dominated by local detail, so a model of them alone babbles. Semantic tokens are few (25 per second) and capture content, so the first stage can plan structure and the acoustic stages render it.

**Question:** A generator produces every clip from one of the two sound classes in the real data, with perfect quality. What happens to FAD?

- A. Nothing, because each clip is realistic
- B. It rises: the fitted mean moves and the covariance shrinks, so both terms grow
- C. It falls, because the generated set is more consistent
- D. It becomes undefined

**Answer:** B. It rises: the fitted mean moves and the covariance shrinks, so both terms grow — FAD compares the whole sets. Dropping a mode shifts $\mu_g$ and removes the between-class variance from $\Sigma_g$. In the figure, generating only class A gives a closed-form FAD of 2.142 (mean term 1.480, covariance term 0.662).

**Key points**

- Generate in a compressed space: codec tokens (discrete) or a VAE latent of the mel spectrogram (continuous).
- AudioLM: w2v-BERT semantic tokens (25 Hz) → SoundStream levels 1–4 → levels 5–12; continuations keep voice and acoustics.
- AudioGen: T5 text embedding + Transformer over codec tokens; AudioLDM: CLAP-conditioned latent diffusion + HiFi-GAN.
- Classifier-free guidance $\tilde\epsilon=\epsilon_\varnothing+w(\epsilon_c-\epsilon_\varnothing)$ trades variety for prompt adherence.
- FAD $=\|\mu_r-\mu_g\|^2+\operatorname{tr}(\Sigma_r+\Sigma_g-2(\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2})^{1/2})$; KL and CLAP score check the prompt.
- Report the embedding model and set sizes with FAD; small sets inflate it.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
