Definition
Audio generation is learning a distribution over waveforms, or for a condition such as a text prompt, and sampling sounds from it: speech continuations, sound effects, ambiences, music. Two families dominate. Token language models generate the discrete codes of a neural codec, as in AudioLM (Borsos et al., 2022) and AudioGen (Kreuk et al., ICLR 2023). Latent diffusion models denoise a compressed spectrogram representation, as in AudioLDM (Liu et al., ICML 2023).
Raw audio is a hard target. One second at 16 kHz is 16,000 samples, and a sound has structure at every scale: sample-level periodicity, phonemes or note onsets every tens of milliseconds, and words, rhythm or scenes over seconds. WaveNet models samples directly. Sampling is slow, and without conditioning it produces speech-like babble with no long-range structure. Modern systems first compress audio into a much shorter representation, either the discrete tokens of a Neural codec models codec or the continuous latent of a VAE over a Mel spectrograms image, and generate in that space.
Tokens: AudioLM and AudioGen
AudioLM uses two tokenizers. Semantic tokens come from k-means (1,024 clusters) on intermediate features of a self-supervised w2v-BERT model, at 25 per second. They capture content and long-range structure but cannot be decoded to good audio. Acoustic tokens are the RVQ codes of SoundStream, which carry speaker, timbre and recording conditions and decode to a waveform. Generation is split into three autoregressive Transformer stages:
- semantic tokens, 25 per second
- SoundStream code of RVQ level at frame (50 frames per second, 12 levels of 1,024 entries)
- the fine stage ignores the semantic tokens; each factor is one decoder-only Transformer
Prompted with a few seconds of speech, AudioLM continues it with the same voice and recording conditions and with syntactically plausible content, without any transcripts in training. AudioGen conditions on text instead. A T5 encoder embeds the description, the Transformer cross-attends to it, and generation runs over the tokens of a 16 kHz codec. AudioGen also mixes pairs of training clips and joins their captions, so the model learns to compose sources ("a dog barks while rain falls").
Latent diffusion: AudioLDM
AudioLDM treats the mel spectrogram as an image. A VAE compresses it to a latent , and a U-Net is trained to remove noise from as in Latent diffusion models. A HiFi-GAN vocoder turns the decoded mel into 16 kHz audio. The condition is a CLAP embedding. CLAP is trained contrastively, like CLIP, so that text and audio embeddings of matching pairs are close. AudioLDM trains with the CLAP embedding of the audio itself and swaps in the embedding of the text prompt at sampling time, so training needs no captions.
- noised latent of the mel spectrogram at diffusion step
- CLAP embedding: of the audio in training, of the text prompt at sampling time
- the condition dropped; training drops it at random so one network learns both terms
- guidance scale. is plain conditional sampling; follows the prompt more closely at the cost of variety. Token models apply the same rule to logits
Later systems keep the recipe and change the parts: larger text encoders, transformer denoisers, or latents of a waveform autoencoder instead of mels. Stable Audio, for example, adds timing conditioning so that the length of the output can be chosen.
The reference systems
- AudioLM
- w2v-BERT semantic tokens (k-means, 1,024) at 25 Hz; SoundStream acoustic tokens, 12 RVQ levels at 50 Hz; three Transformer stages (semantic, coarse levels 1–4, fine levels 5–12); speech trained on LibriLight without transcripts
- AudioGen
- text → T5 encoder → autoregressive Transformer over 16 kHz codec tokens; classifier-free guidance on logits; caption-mixing augmentation
- AudioLDM
- VAE over mel spectrograms, latent diffusion U-Net conditioned on CLAP, HiFi-GAN vocoder, 16 kHz; evaluated on AudioCaps
- FAD
- Kilgour et al., Interspeech 2019: Fréchet distance between Gaussians fitted to 128-d VGGish embeddings of real and generated sets
- KL and CLAP score
- KL between a pretrained tagger's label distributions for reference and generated clips of the same prompt; CLAP score: cosine between the prompt's and the clip's CLAP embeddings
Evaluation
There is no single target waveform for "rain on a tin roof", so metrics compare distributions or check adherence. FAD embeds every real and every generated clip, fits a Gaussian to each set and computes the Fréchet distance between them. Lower is better:
- mean and covariance of the real clips' embeddings
- the same for generated clips
- zero when the covariances are equal; grows when the generator has too little or too much variety
FAD needs no pairing, but it says nothing about whether a clip matches its prompt. KL runs a pretrained audio tagger (for example PANNs or PaSST) on the reference and the generated clip for the same prompt and compares the label distributions. CLAP score measures text–audio agreement directly. Listening tests with overall-quality and text-relevance ratings remain the final check.
Worked example: FAD by hand
- 1Real embeddings: , . Generated: , , so the generator is shifted and too narrow in the first dimension.
- 2Mean term: .
- 3Diagonal covariances commute, so .
- 4Covariance term: .
- 5. The two failures cost the same here; per dimension the covariance term is .
1import numpy as np23def frechet_audio_distance(emb_real, emb_gen):4 """FAD between two sets of clip embeddings, shape (n_clips, dim), e.g. 128-d VGGish."""5 mu_r, mu_g = emb_real.mean(0), emb_gen.mean(0)6 S_r, S_g = np.cov(emb_real, rowvar=False), np.cov(emb_gen, rowvar=False)7 w, V = np.linalg.eigh(S_r)8 h = (V * np.sqrt(np.clip(w, 0, None))) @ V.T # S_r^(1/2)9 cross = np.sqrt(np.clip(np.linalg.eigvalsh(h @ S_g @ h), 0, None)).sum()10 return float(((mu_r - mu_g) ** 2).sum() + np.trace(S_r) + np.trace(S_g) - 2 * cross)1112rng = np.random.default_rng(0)13real = rng.normal(size=(2000, 128))14print(round(frechet_audio_distance(real, rng.normal(size=(2000, 128))), 2)) # 4.24: sampling floor15print(round(frechet_audio_distance(real, rng.normal(size=(2000, 128)) + 0.2), 2)) # 9.35: shifted16print(round(frechet_audio_distance(real, 0.7 * rng.normal(size=(2000, 128))), 2)) # 14.52: too little variety1718# Sampling a text-to-audio model (diffusers, not run here: needs torch and a download):19# from diffusers import AudioLDMPipeline20# pipe = AudioLDMPipeline.from_pretrained("cvssp/audioldm-s-full-v2")21# audio = pipe("rain on a tin roof", num_inference_steps=10, audio_length_in_s=5.0).audios[0] # 16 kHzUse when
- +Token LMs: one Transformer recipe for speech, sounds and music; prompting with audio continues voice and acoustics; streaming is natural.
- +Latent diffusion: parallel generation of the whole clip, fewer sequential steps, easy inpainting and editing in the latent.
Avoid when
- −Token LMs: thousands of tokens per second of audio across codebooks; sampling is sequential and errors can compound.
- −Latent diffusion: fixed-length windows, quality capped by the VAE and vocoder; many denoising steps unless distilled.
Check yourself
Why does AudioLM use semantic tokens in addition to SoundStream tokens?
Check yourself
A generator produces every clip from one of the two sound classes in the real data, with perfect quality. What happens to FAD?
Remember
- ✓Generate in a compressed space: codec tokens (discrete) or a VAE latent of the mel spectrogram (continuous).
- ✓AudioLM: w2v-BERT semantic tokens (25 Hz) → SoundStream levels 1–4 → levels 5–12; continuations keep voice and acoustics.
- ✓AudioGen: T5 text embedding + Transformer over codec tokens; AudioLDM: CLAP-conditioned latent diffusion + HiFi-GAN.
- ✓Classifier-free guidance trades variety for prompt adherence.
- ✓FAD ; KL and CLAP score check the prompt.
- ✓Report the embedding model and set sizes with FAD; small sets inflate it.