Audio generation

General audio generators either model codec tokens autoregressively (AudioLM, AudioGen) or run latent diffusion over mel latents (AudioLDM), steered by T5 or CLAP text embeddings and classifier-free guidance, and are scored by FAD, KL and CLAP score.

8 min read729 wordsadvanced0 viewsupdated 2026-10-15

Definition

Audio generation is learning a distribution over waveforms, p(x)p(x) or p(x∣c)p(x\mid c) for a condition cc such as a text prompt, and sampling sounds from it: speech continuations, sound effects, ambiences, music. Two families dominate. Token language models generate the discrete codes of a neural codec, as in AudioLM (Borsos et al., 2022) and AudioGen (Kreuk et al., ICLR 2023). Latent diffusion models denoise a compressed spectrogram representation, as in AudioLDM (Liu et al., ICML 2023).

Raw audio is a hard target. One second at 16 kHz is 16,000 samples, and a sound has structure at every scale: sample-level periodicity, phonemes or note onsets every tens of milliseconds, and words, rhythm or scenes over seconds. WaveNet models samples directly. Sampling is slow, and without conditioning it produces speech-like babble with no long-range structure. Modern systems first compress audio into a much shorter representation, either the discrete tokens of a Neural codec models codec or the continuous latent of a VAE over a Mel spectrograms image, and generate in that space.

Tokens: AudioLM and AudioGen

AudioLM uses two tokenizers. Semantic tokens come from k-means (1,024 clusters) on intermediate features of a self-supervised w2v-BERT model, at 25 per second. They capture content and long-range structure but cannot be decoded to good audio. Acoustic tokens are the RVQ codes of SoundStream, which carry speaker, timbre and recording conditions and decode to a waveform. Generation is split into three autoregressive Transformer stages:

p(x)  ≈  p(s1:Ts)⏟semantic  p(a1:Ta1:4∣s)⏟coarse acoustic  p(a1:Ta5:12∣a1:4)⏟fine acousticp(x)\;\approx\;\underbrace{p(s_{1:T_s})}_{\text{semantic}}\;\underbrace{p\big(a^{1:4}_{1:T_a}\mid s\big)}_{\text{coarse acoustic}}\;\underbrace{p\big(a^{5:12}_{1:T_a}\mid a^{1:4}\big)}_{\text{fine acoustic}}
s1:Tss_{1:T_s}
semantic tokens, 25 per second
atqa^{q}_{t}
SoundStream code of RVQ level qq at frame tt (50 frames per second, 12 levels of 1,024 entries)
≈\approx
the fine stage ignores the semantic tokens; each factor is one decoder-only Transformer
A · language model over discrete tokens (AudioLM, Borsos et al. 2022)w2v-BERTlayer featuresk-means, 1024→ semantic1 · semantic, 25 Hzlong-term structure:words, melody2 · coarse acousticRVQ levels 1–43 · fine acousticlevels 5–12 → SoundStreameach stage: a decoder-only Transformer conditioned on all earlier tokensAudioGen (Kreuk et al.): text → T5 encoder → one Transformer over codec tokensB · latent diffusion (AudioLDM, Liu et al. 2023)CLAPaudio emb. (train)text emb. (sample)one joint spacedenoiser (U-Net)z_T ~ N(0, I)→ … → z_0, T stepscondition: CLAP emb.VAEdecoderlatent →melmelHiFi-GANvocoder→ 16 kHzwaveformCLAP: one space for text and audio → train on audio alone, prompt with textshared toolsclassifier-free guidance: ε̃ = ε(z, ∅) + w · (ε(z, c) − ε(z, ∅))same rule on logits for token models · train with the condition dropped at randomFAD ↓Fréchet distance ofGaussians fitted toVGGish embeddingssets, no pairingKL ↓audio classifier'slabel distribution:reference vs generatedpaired per promptCLAP score ↑cosine between CLAPtext and audioembeddingsprompt adherence
fig.The two families. A: AudioLM generates semantic tokens, then the first 4 SoundStream RVQ levels, then levels 5–12, and the codec decodes the result; AudioGen replaces the semantic stage with a text encoder. B: AudioLDM denoises a VAE latent of a mel spectrogram under a CLAP embedding, then a VAE decoder and HiFi-GAN produce the waveform. Bottom: classifier-free guidance and the three standard metrics.

Prompted with a few seconds of speech, AudioLM continues it with the same voice and recording conditions and with syntactically plausible content, without any transcripts in training. AudioGen conditions on text instead. A T5 encoder embeds the description, the Transformer cross-attends to it, and generation runs over the tokens of a 16 kHz codec. AudioGen also mixes pairs of training clips and joins their captions, so the model learns to compose sources ("a dog barks while rain falls").

Latent diffusion: AudioLDM

AudioLDM treats the mel spectrogram as an image. A VAE compresses it to a latent z0z_0, and a U-Net is trained to remove noise from ztz_t as in Latent diffusion models. A HiFi-GAN vocoder turns the decoded mel into 16 kHz audio. The condition is a CLAP embedding. CLAP is trained contrastively, like CLIP, so that text and audio embeddings of matching pairs are close. AudioLDM trains with the CLAP embedding of the audio itself and swaps in the embedding of the text prompt at sampling time, so training needs no captions.

L=Ez0,ϵ,t∥ϵ−ϵθ(zt,t,E)∥2,ϵ~=ϵθ(zt,t,∅)+w(ϵθ(zt,t,E)−ϵθ(zt,t,∅))\mathcal L=\mathbb E_{z_0,\epsilon,t}\big\|\epsilon-\epsilon_\theta(z_t,t,E)\big\|^2,\qquad \tilde\epsilon=\epsilon_\theta(z_t,t,\varnothing)+w\big(\epsilon_\theta(z_t,t,E)-\epsilon_\theta(z_t,t,\varnothing)\big)
ztz_t
noised latent of the mel spectrogram at diffusion step tt
EE
CLAP embedding: of the audio in training, of the text prompt at sampling time
∅\varnothing
the condition dropped; training drops it at random so one network learns both terms
ww
guidance scale. w=1w=1 is plain conditional sampling; w>1w>1 follows the prompt more closely at the cost of variety. Token models apply the same rule to logits

Later systems keep the recipe and change the parts: larger text encoders, transformer denoisers, or latents of a waveform autoencoder instead of mels. Stable Audio, for example, adds timing conditioning so that the length of the output can be chosen.

The reference systems

AudioLM
w2v-BERT semantic tokens (k-means, 1,024) at 25 Hz; SoundStream acoustic tokens, 12 RVQ levels at 50 Hz; three Transformer stages (semantic, coarse levels 1–4, fine levels 5–12); speech trained on LibriLight without transcripts
AudioGen
text → T5 encoder → autoregressive Transformer over 16 kHz codec tokens; classifier-free guidance on logits; caption-mixing augmentation
AudioLDM
VAE over mel spectrograms, latent diffusion U-Net conditioned on CLAP, HiFi-GAN vocoder, 16 kHz; evaluated on AudioCaps
FAD
Kilgour et al., Interspeech 2019: Fréchet distance between Gaussians fitted to 128-d VGGish embeddings of real and generated sets
KL and CLAP score
KL between a pretrained tagger's label distributions for reference and generated clips of the same prompt; CLAP score: cosine between the prompt's and the clip's CLAP embeddings

Evaluation

There is no single target waveform for "rain on a tin roof", so metrics compare distributions or check adherence. FAD embeds every real and every generated clip, fits a Gaussian to each set and computes the Fréchet distance between them. Lower is better:

FAD=∥μr−μg∥2+tr⁡ ⁣(Σr+Σg−2(Σr1/2ΣgΣr1/2)1/2)\mathrm{FAD}=\big\|\mu_r-\mu_g\big\|^2+\operatorname{tr}\!\Big(\Sigma_r+\Sigma_g-2\big(\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2}\big)^{1/2}\Big)
μr,Σr\mu_r,\Sigma_r
mean and covariance of the real clips' embeddings
μg,Σg\mu_g,\Sigma_g
the same for generated clips
tr⁡(⋅)\operatorname{tr}(\cdot)
zero when the covariances are equal; grows when the generator has too little or too much variety

FAD needs no pairing, but it says nothing about whether a clip matches its prompt. KL runs a pretrained audio tagger (for example PANNs or PaSST) on the reference and the generated clip for the same prompt and compares the label distributions. CLAP score measures text–audio agreement directly. Listening tests with overall-quality and text-relevance ratings remain the final check.

Fréchet audio distance · 4-D toy embeddings · 200 real and 200 generated clipsdims 1–2 of 4 · ellipses: fitted Gaussians (2σ)● real: two sound classes, 50 / 50 · ● generatedterms: 200 samples / closed form (N → ∞)mean term ‖μr − μg‖²0.313 / 0.250covariance term0.042 / 0.000FAD0.355 / 0.250floor, real vs real (200 each): 0.086closed-form FAD vs shift (max 2.25)−1.5+1.5FAD = ‖μr − μg‖² + tr(Σr + Σg − 2(ΣrΣg)^½) = 0.355a shift of the whole distribution moves only the mean termreal FAD uses e.g. 128-d VGGish embeddings; with few clips the estimate is biased upwards
clips per set
fig.FAD between toy 4-D embeddings: real clips come from two classes, the generator is the same mixture with a shift, a spread factor and a class share. Default (shift 0.5, 200 clips per set): FAD 0.355 from samples against 0.250 in closed form, all of it in the mean term; two real sets of 200 already give 0.086. With 50 clips an exact copy of the real distribution scores 0.814, and with 1,000 it scores 0.017. Spread ×0.5 moves only the covariance term (closed form 0.505); generating only class A gives 2.142 in closed form, split 1.480 / 0.662.

Worked example: FAD by hand

  1. 1Real embeddings: μr=(0,0)\mu_r=(0,0), Σr=diag(1,1)\Sigma_r=\mathrm{diag}(1,1). Generated: μg=(0.5,0)\mu_g=(0.5,0), Σg=diag(0.25,1)\Sigma_g=\mathrm{diag}(0.25,1), so the generator is shifted and too narrow in the first dimension.
  2. 2Mean term: ∥μr−μg∥2=0.52+02=0.25\|\mu_r-\mu_g\|^2=0.5^2+0^2=0.25.
  3. 3Diagonal covariances commute, so (Σr1/2ΣgΣr1/2)1/2=diag(0.25,1)=diag(0.5,1)(\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2})^{1/2}=\mathrm{diag}(\sqrt{0.25},\sqrt1)=\mathrm{diag}(0.5,1).
  4. 4Covariance term: tr⁡Σr+tr⁡Σg−2⋅1.5=2+1.25−3=0.25\operatorname{tr}\Sigma_r+\operatorname{tr}\Sigma_g-2\cdot1.5=2+1.25-3=0.25.
  5. 5FAD=0.25+0.25=0.5\mathrm{FAD}=0.25+0.25=0.5. The two failures cost the same here; per dimension the covariance term is (σr−σg)2=(1−0.5)2(\sigma_r-\sigma_g)^2=(1-0.5)^2.
python
1import numpy as np
2
3def frechet_audio_distance(emb_real, emb_gen):
4 """FAD between two sets of clip embeddings, shape (n_clips, dim), e.g. 128-d VGGish."""
5 mu_r, mu_g = emb_real.mean(0), emb_gen.mean(0)
6 S_r, S_g = np.cov(emb_real, rowvar=False), np.cov(emb_gen, rowvar=False)
7 w, V = np.linalg.eigh(S_r)
8 h = (V * np.sqrt(np.clip(w, 0, None))) @ V.T # S_r^(1/2)
9 cross = np.sqrt(np.clip(np.linalg.eigvalsh(h @ S_g @ h), 0, None)).sum()
10 return float(((mu_r - mu_g) ** 2).sum() + np.trace(S_r) + np.trace(S_g) - 2 * cross)
11
12rng = np.random.default_rng(0)
13real = rng.normal(size=(2000, 128))
14print(round(frechet_audio_distance(real, rng.normal(size=(2000, 128))), 2)) # 4.24: sampling floor
15print(round(frechet_audio_distance(real, rng.normal(size=(2000, 128)) + 0.2), 2)) # 9.35: shifted
16print(round(frechet_audio_distance(real, 0.7 * rng.normal(size=(2000, 128))), 2)) # 14.52: too little variety
17
18# Sampling a text-to-audio model (diffusers, not run here: needs torch and a download):
19# from diffusers import AudioLDMPipeline
20# pipe = AudioLDMPipeline.from_pretrained("cvssp/audioldm-s-full-v2")
21# audio = pipe("rain on a tin roof", num_inference_steps=10, audio_length_in_s=5.0).audios[0] # 16 kHz
FAD from two embedding matrices (numpy, runs as is; in practice the rows are VGGish or CLAP embeddings of real and generated clips)

Use when

  • +Token LMs: one Transformer recipe for speech, sounds and music; prompting with audio continues voice and acoustics; streaming is natural.
  • +Latent diffusion: parallel generation of the whole clip, fewer sequential steps, easy inpainting and editing in the latent.

Avoid when

  • −Token LMs: thousands of tokens per second of audio across codebooks; sampling is sequential and errors can compound.
  • −Latent diffusion: fixed-length windows, quality capped by the VAE and vocoder; many denoising steps unless distilled.

Check yourself

Why does AudioLM use semantic tokens in addition to SoundStream tokens?

Check yourself

A generator produces every clip from one of the two sound classes in the real data, with perfect quality. What happens to FAD?

Remember

  • ✓Generate in a compressed space: codec tokens (discrete) or a VAE latent of the mel spectrogram (continuous).
  • ✓AudioLM: w2v-BERT semantic tokens (25 Hz) → SoundStream levels 1–4 → levels 5–12; continuations keep voice and acoustics.
  • ✓AudioGen: T5 text embedding + Transformer over codec tokens; AudioLDM: CLAP-conditioned latent diffusion + HiFi-GAN.
  • ✓Classifier-free guidance ϵ~=ϵ∅+w(ϵc−ϵ∅)\tilde\epsilon=\epsilon_\varnothing+w(\epsilon_c-\epsilon_\varnothing) trades variety for prompt adherence.
  • ✓FAD =∥μr−μg∥2+tr⁡(Σr+Σg−2(Σr1/2ΣgΣr1/2)1/2)=\|\mu_r-\mu_g\|^2+\operatorname{tr}(\Sigma_r+\Sigma_g-2(\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2})^{1/2}); KL and CLAP score check the prompt.
  • ✓Report the embedding model and set sizes with FAD; small sets inflate it.
← back to Deep Learning roadmap