# Neural codec models

> Neural codecs (SoundStream 2021, EnCodec 2022) compress audio with a conv encoder, residual vector quantisation and a GAN-trained decoder; at 24 kHz, 75 frames/s × N codebooks × 10 bits gives 1.5–24 kbps, and the codes are tokens for audio LMs.

Source: https://www.mlfoundry.in/learn/neural-codec-models  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 8 min read · updated 2026-10-15

**Neural audio codec:** is an encoder–quantiser–decoder network trained end to end to turn a waveform into a short stream of discrete codes and back. The two reference designs are **SoundStream** (Zeghidour, Luebs, Omran, Skoglund and Tagliasacchi, "SoundStream: An End-to-End Neural Audio Codec", 2021, IEEE/ACM TASLP) and **EnCodec** (Défossez, Copet, Synnaeve and Adi, "High Fidelity Neural Audio Compression", 2022). Both use a convolutional encoder, **residual vector quantisation** (RVQ) and a decoder trained with reconstruction and adversarial losses.

The task: take 24,000 samples per second of 16-bit audio (384 kbps) and send a few kilobits per second, so that the decoded audio still sounds like the original. Classic codecs such as Opus and EVS do this with hand-designed transforms and psychoacoustic models. At very low rates their output becomes muffled and buzzy. A neural codec learns the transform and the decoder from data, and its adversarial loss lets the decoder produce plausible detail that was never transmitted.

The discrete codes turned out to matter as much as the compression. A codec maps audio to a sequence of integers drawn from small vocabularies, so audio can be modelled by the same Transformers that model text. See **Audio generation**, **Music generation** and the codec language models in **Text-to-speech**.

## Encoder, quantiser, decoder

At 24 kHz both encoders start with a 7-tap convolution and then apply four residual blocks with strides 2, 4, 5 and 8. The total stride is $M=320$, so there are $24000/320=75$ latent frames per second. The channels double at every downsampling. EnCodec adds a two-layer LSTM and a final convolution to $D=128$ dimensions. The decoder mirrors the encoder with transposed convolutions. SoundStream uses only causal convolutions, so it can stream with low latency.

> Figure: A neural codec at 24 kHz. The encoder (strides 2·4·5·8 = 320) turns 24,000 samples into 75 latent frames per second. RVQ quantises each frame in N stages, each stage coding the residual left by the previous ones and emitting one 10-bit index from a 1024-entry codebook. The decoder reconstructs from the sum of the chosen code vectors, and training combines reconstruction, adversarial, feature-matching and commitment losses.

A single vector quantiser cannot reach useful rates. At 6 kbps each frame gets $6000/75=80$ bits, which would need a codebook of $2^{80}$ vectors. RVQ uses a cascade of $N_q$ small codebooks instead. The first stage quantises the frame, and every later stage quantises the error that is left:

$$
r_0=z,\qquad q_i=e^{(i)}_{k_i},\;\;k_i=\arg\min_k\big\|r_{i-1}-e^{(i)}_k\big\|,\qquad r_i=r_{i-1}-q_i,\qquad \hat z=\sum_{i=1}^{N_q}q_i
$$

- $z$: one encoder frame ($D$-dimensional)
- $e^{(i)}_k$: entry $k$ of codebook $i$, $K=1024$ entries each in SoundStream and EnCodec at 24 kHz
- $k_i$: the transmitted index of stage $i$, $\log_2K=10$ bits
- $\hat z$: what the decoder receives; dropping the last stages gives a coarser $\hat z$

$$
R=\frac{f_s}{M}\cdot N_q\cdot\log_2 K\qquad\text{e.g. }\;\frac{24000}{320}\cdot 8\cdot 10=6000\ \text{bit/s}
$$

- $f_s/M$: latent frame rate: sample rate over total encoder stride
- $N_q$: number of codebooks actually sent
- $\log_2K$: bits per index

Codebooks are learned as in **Vector-quantized VAE (VQ-VAE)**: the code vectors follow exponential moving averages of the frames assigned to them, a commitment loss pulls the encoder output towards the chosen codes, and gradients pass through the quantiser with the straight-through estimator. SoundStream initialises the codebooks with k-means on the first batch and replaces codes that go unused.

## Losses and quantizer dropout

$$
\mathcal L_G=\lambda_t\|x-\hat x\|_1+\lambda_f\sum_{s}\big\|\mathcal S_s(x)-\mathcal S_s(\hat x)\big\|+\lambda_g\mathcal L_{\text{adv}}+\lambda_{\text{feat}}\mathcal L_{\text{feat}}+\lambda_w\sum_{i}\big\|r_{i-1}-q_i\big\|_2^2
$$

- $\mathcal S_s$: mel spectrogram at scale $s$; EnCodec combines L1 and L2 over several window sizes
- $\mathcal L_{\text{adv}}$: generator term of the **Adversarial loss**; EnCodec uses a multi-scale STFT discriminator, SoundStream a waveform and an STFT discriminator
- $\mathcal L_{\text{feat}}$: L1 distance between discriminator features of $x$ and $\hat x$
- $\lambda_w$: weight of the commitment term, summed over the RVQ stages

The reconstruction terms alone give dull, over-smoothed audio. The discriminators reward realistic fine structure, as in the GAN vocoders (**HiFi-GAN**). EnCodec also adds a loss balancer, which rescales each loss's gradient so that its weight sets its share of the total gradient, independent of its raw scale.

**Quantizer dropout** makes one model serve many bitrates. For each training example, SoundStream draws $n_q$ uniformly from $1,\dots,N_q$ and uses only the first $n_q$ stages. The decoder therefore learns to reconstruct from any prefix of the codes, and the sender picks the rate at inference by truncating. The paper reports that one such model matches models trained separately for each bitrate.

**Published configurations**

- **SoundStream:** 24 kHz, 75 frames/s, 1024-entry codebooks; one model for 3–18 kbps with quantizer dropout; at 3 kbps it beat Opus at 12 kbps and approached EVS at 9.6 kbps in listening tests; runs in real time on a smartphone CPU
- **EnCodec 24 kHz:** mono, 1.5 / 3 / 6 / 12 / 24 kbps = 2 / 4 / 8 / 16 / 32 codebooks of 1024 at 75 frames/s
- **EnCodec 48 kHz:** stereo model at 3 / 6 / 12 / 24 kbps; an optional small Transformer entropy model compresses the codes further
- **EnCodec 32 kHz:** the tokenizer of MusicGen: 4 codebooks of 2048 at 50 frames/s = 2.2 kbps
- **DAC (Kumar et al., 2023):** 44.1 kHz audio at about 8 kbps with 9 codebooks; codes looked up in a low-dimensional, L2-normalised space to keep all entries in use

> Figure: RVQ trained by k-means stage by stage on 512 toy 2-D frames and scored on 512 held-out ones. Default (K = 8, 2 stages): held-out SNR 19.6 dB, close to the 6.02 dB-per-bit rule (2 × 9.03 = 18.1 dB). Each extra codebook adds less: with 8 stages the held-out SNR stalls at 38.4 dB while training SNR reaches 58.5 dB, because late codebooks fit noise. K = 32 gives 17.6 dB from one stage. The calculator reproduces 6 kbps = 600 tokens/s for EnCodec 24 kHz with 8 codebooks (64× below 16-bit PCM), 2.2 kbps for MusicGen's codec and 7.75 kbps for DAC.

## Worked example: ten seconds through EnCodec 24 kHz

240,000 samples → 750 frames. Bits for 10 s and compression against 16-bit PCM at 384 kbps

| Bandwidth | Codebooks | Tokens per s | Tokens in 10 s | Bits in 10 s | vs PCM |
| --- | --- | --- | --- | --- | --- |
| 1.5 kbps | 2 | 150 | 1,500 | 15,000 | 256× |
| 3 kbps | 4 | 300 | 3,000 | 30,000 | 128× |
| 6 kbps | 8 | 600 | 6,000 | 60,000 | 64× |
| 12 kbps | 16 | 1,200 | 12,000 | 120,000 | 32× |
| 24 kbps | 32 | 2,400 | 24,000 | 240,000 | 16× |

At 6 kbps the 10-second clip is 7,500 bytes, against 480,000 bytes of PCM. For a language model it is also 6,000 tokens, far more than the words spoken in it. That length is why audio LMs split the codebooks into coarse and fine stages or predict several codebooks per step.

```python
import numpy as np
from transformers import AutoProcessor, EncodecModel

model = EncodecModel.from_pretrained("facebook/encodec_24khz")
processor = AutoProcessor.from_pretrained("facebook/encodec_24khz")

sr = processor.sampling_rate                        # 24000
t = np.arange(10 * sr) / sr
audio = 0.3 * np.sin(2 * np.pi * 220 * t) * (1 + np.sin(2 * np.pi * 2 * t)) / 2

inputs = processor(raw_audio=audio, sampling_rate=sr, return_tensors="pt")
enc = model.encode(inputs["input_values"], inputs["padding_mask"], bandwidth=6.0)
print(enc.audio_codes.shape)                        # (1, 1, 8, 750): chunks, batch, codebooks, frames
print(enc.audio_codes.max() < 1024)                 # every token is a 10-bit index

recon = model.decode(enc.audio_codes, enc.audio_scales, inputs["padding_mask"])[0]
bits = enc.audio_codes.numel() * np.log2(1024)
print(bits / 10, "bit/s")                           # 6000.0
```

Encode and decode with EnCodec through Hugging Face transformers (not run here: needs torch and a model download)

> **Bitrate is not quality, and SNR is not either:** The adversarial decoder is not trying to reproduce the waveform sample by sample. It produces audio that sounds right, so waveform SNR of a neural codec is often low even when listeners rate it highly. Codecs are compared with listening tests (MUSHRA) and perceptual metrics such as ViSQOL, at a stated bitrate and sample rate.

> **Codebook order is a coarse-to-fine hierarchy:** Because each stage codes what the previous ones missed, the first codebook carries most of the energy and much of the content, and later ones add detail. Audio LMs exploit this. VALL-E predicts the first of EnCodec's 8 codebooks autoregressively and the other 7 with a non-autoregressive model, and AudioLM generates coarse codebooks before fine ones.

**Question:** Why do SoundStream and EnCodec use residual VQ instead of one large codebook?

- A. A single codebook cannot be trained with the straight-through estimator
- B. At 6 kbps a frame carries 80 bits, so one codebook would need 2^80 entries; 8 stages of 1024 entries reach the same rate with 8,192 vectors
- C. RVQ removes the need for an adversarial loss
- D. A single codebook would make the frame rate too high

**Answer:** B. At 6 kbps a frame carries 80 bits, so one codebook would need 2^80 entries; 8 stages of 1024 entries reach the same rate with 8,192 vectors — The rate per frame is $N_q\log_2K$ bits. RVQ makes the number of stored vectors grow linearly ($N_qK$) instead of exponentially ($K^{N_q}$) in the bits per frame, and searching it costs $N_qK$ distance computations.

**Question:** What does quantizer dropout buy?

- A. Robustness to packet loss
- B. A smaller encoder
- C. One model that decodes from any prefix of the codebooks, so the bitrate is chosen at inference by sending fewer codes
- D. Better codebook usage

**Answer:** C. One model that decodes from any prefix of the codebooks, so the bitrate is chosen at inference by sending fewer codes — Training with a random number of stages forces the decoder to work from the first $n_q$ codes for every $n_q$. SoundStream covers 3–18 kbps with one model this way.

**Key points**

- Conv encoder with strides 2·4·5·8 = 320: 24 kHz audio → 75 latent frames per second.
- RVQ: each of $N_q$ codebooks quantises the residual of the previous ones; $\hat z=\sum_i q_i$.
- Bitrate $R=(f_s/M)\,N_q\log_2K$: EnCodec 24 kHz gives 1.5–24 kbps with 2–32 codebooks of 1024.
- Training: waveform + multi-scale mel reconstruction, adversarial and feature-matching losses, commitment loss.
- Quantizer dropout trains one model for every bitrate; the first codebooks carry the coarse content.
- The codes are the vocabulary of audio LMs: AudioLM (SoundStream), VALL-E (EnCodec 8 codebooks), MusicGen (EnCodec 32 kHz, 4 codebooks).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
