Neural codec models

Neural codecs (SoundStream 2021, EnCodec 2022) compress audio with a conv encoder, residual vector quantisation and a GAN-trained decoder; at 24 kHz, 75 frames/s × N codebooks × 10 bits gives 1.5–24 kbps, and the codes are tokens for audio LMs.

8 min read743 wordsadvanced0 viewsupdated 2026-10-15

Definition

Neural audio codec is an encoder–quantiser–decoder network trained end to end to turn a waveform into a short stream of discrete codes and back. The two reference designs are SoundStream (Zeghidour, Luebs, Omran, Skoglund and Tagliasacchi, "SoundStream: An End-to-End Neural Audio Codec", 2021, IEEE/ACM TASLP) and EnCodec (Défossez, Copet, Synnaeve and Adi, "High Fidelity Neural Audio Compression", 2022). Both use a convolutional encoder, residual vector quantisation (RVQ) and a decoder trained with reconstruction and adversarial losses.

The task: take 24,000 samples per second of 16-bit audio (384 kbps) and send a few kilobits per second, so that the decoded audio still sounds like the original. Classic codecs such as Opus and EVS do this with hand-designed transforms and psychoacoustic models. At very low rates their output becomes muffled and buzzy. A neural codec learns the transform and the decoder from data, and its adversarial loss lets the decoder produce plausible detail that was never transmitted.

The discrete codes turned out to matter as much as the compression. A codec maps audio to a sequence of integers drawn from small vocabularies, so audio can be modelled by the same Transformers that model text. See Audio generation, Music generation and the codec language models in Text-to-speech.

Encoder, quantiser, decoder

At 24 kHz both encoders start with a 7-tap convolution and then apply four residual blocks with strides 2, 4, 5 and 8. The total stride is M=320M=320, so there are 24000/320=7524000/320=75 latent frames per second. The channels double at every downsampling. EnCodec adds a two-layer LSTM and a final convolution to D=128D=128 dimensions. The decoder mirrors the encoder with transposed convolutions. SoundStream uses only causal convolutions, so it can stream with low latency.

SoundStream / EnCodec at 24 kHz · numbers per second of audiox · 24 kHz24,000 samplesconv encoder (causal in SoundStream)conv k7 → 4 blocks, strides 2·4·5·8= 320× downsampling, channels ×2 eachEnCodec: + 2-layer LSTM, conv → D = 128z: 75 frames × Dresidual vector quantisation, per framer₀ = zQ₁K = 1024r₁q₁, c₁Q₂K = 1024r₂q₂, c₂Q₃K = 1024r₃ …q₃, c₃Q_NK = 1024q_N, c_Nrᵢ = rᵢ₋₁ − qᵢc₁5128790341660… 75 per sc₂127713059986… 75 per sc₃43021964587812… 75 per s10 bitseachbitrate = 75 × N × 10N = 2 → 1.5 kbpsN = 8 → 6 kbps · N = 32 → 24ẑ = q₁ + … + q_Ndecodermirror: transposed convsstrides 8·5·4·2 → 24 kHzx̂training lossesreconstruction: wave L1 + multi-scale meladversarial: STFT / wave discriminatorsfeature matching · commitment ‖z − ẑ‖²quantizer dropout: random N per batchcodes c₁ … c_N are the tokens that audio languagemodels such as AudioLM, VALL-E and MusicGen predict
fig.A neural codec at 24 kHz. The encoder (strides 2·4·5·8 = 320) turns 24,000 samples into 75 latent frames per second. RVQ quantises each frame in N stages, each stage coding the residual left by the previous ones and emitting one 10-bit index from a 1024-entry codebook. The decoder reconstructs from the sum of the chosen code vectors, and training combines reconstruction, adversarial, feature-matching and commitment losses.

A single vector quantiser cannot reach useful rates. At 6 kbps each frame gets 6000/75=806000/75=80 bits, which would need a codebook of 2802^{80} vectors. RVQ uses a cascade of NqN_q small codebooks instead. The first stage quantises the frame, and every later stage quantises the error that is left:

r0=z,qi=eki(i),    ki=arg⁡min⁡k∥ri−1−ek(i)∥,ri=ri−1−qi,z^=∑i=1Nqqir_0=z,\qquad q_i=e^{(i)}_{k_i},\;\;k_i=\arg\min_k\big\|r_{i-1}-e^{(i)}_k\big\|,\qquad r_i=r_{i-1}-q_i,\qquad \hat z=\sum_{i=1}^{N_q}q_i
zz
one encoder frame (DD-dimensional)
ek(i)e^{(i)}_k
entry kk of codebook ii, K=1024K=1024 entries each in SoundStream and EnCodec at 24 kHz
kik_i
the transmitted index of stage ii, log⁡2K=10\log_2K=10 bits
z^\hat z
what the decoder receives; dropping the last stages gives a coarser z^\hat z
R=fsM⋅Nq⋅log⁡2Ke.g.   24000320⋅8⋅10=6000 bit/sR=\frac{f_s}{M}\cdot N_q\cdot\log_2 K\qquad\text{e.g. }\;\frac{24000}{320}\cdot 8\cdot 10=6000\ \text{bit/s}
fs/Mf_s/M
latent frame rate: sample rate over total encoder stride
NqN_q
number of codebooks actually sent
log⁡2K\log_2K
bits per index

Codebooks are learned as in Vector-quantized VAE (VQ-VAE): the code vectors follow exponential moving averages of the frames assigned to them, a commitment loss pulls the encoder output towards the chosen codes, and gradients pass through the quantiser with the straight-through estimator. SoundStream initialises the codebooks with k-means on the first batch and replaces codes that go unused.

Losses and quantizer dropout

LG=λt∥x−x^∥1+λf∑s∥Ss(x)−Ss(x^)∥+λgLadv+λfeatLfeat+λw∑i∥ri−1−qi∥22\mathcal L_G=\lambda_t\|x-\hat x\|_1+\lambda_f\sum_{s}\big\|\mathcal S_s(x)-\mathcal S_s(\hat x)\big\|+\lambda_g\mathcal L_{\text{adv}}+\lambda_{\text{feat}}\mathcal L_{\text{feat}}+\lambda_w\sum_{i}\big\|r_{i-1}-q_i\big\|_2^2
Ss\mathcal S_s
mel spectrogram at scale ss; EnCodec combines L1 and L2 over several window sizes
Ladv\mathcal L_{\text{adv}}
generator term of the Adversarial loss; EnCodec uses a multi-scale STFT discriminator, SoundStream a waveform and an STFT discriminator
Lfeat\mathcal L_{\text{feat}}
L1 distance between discriminator features of xx and x^\hat x
λw\lambda_w
weight of the commitment term, summed over the RVQ stages

The reconstruction terms alone give dull, over-smoothed audio. The discriminators reward realistic fine structure, as in the GAN vocoders (HiFi-GAN). EnCodec also adds a loss balancer, which rescales each loss's gradient so that its weight sets its share of the total gradient, independent of its raw scale.

Quantizer dropout makes one model serve many bitrates. For each training example, SoundStream draws nqn_q uniformly from 1,…,Nq1,\dots,N_q and uses only the first nqn_q stages. The decoder therefore learns to reconstruct from any prefix of the codes, and the sender picks the rate at inference by truncating. The paper reports that one such model matches models trained separately for each bitrate.

Published configurations

SoundStream
24 kHz, 75 frames/s, 1024-entry codebooks; one model for 3–18 kbps with quantizer dropout; at 3 kbps it beat Opus at 12 kbps and approached EVS at 9.6 kbps in listening tests; runs in real time on a smartphone CPU
EnCodec 24 kHz
mono, 1.5 / 3 / 6 / 12 / 24 kbps = 2 / 4 / 8 / 16 / 32 codebooks of 1024 at 75 frames/s
EnCodec 48 kHz
stereo model at 3 / 6 / 12 / 24 kbps; an optional small Transformer entropy model compresses the codes further
EnCodec 32 kHz
the tokenizer of MusicGen: 4 codebooks of 2048 at 50 frames/s = 2.2 kbps
DAC (Kumar et al., 2023)
44.1 kHz audio at about 8 kbps with 9 codebooks; codes looked up in a low-dimensional, L2-normalised space to keep all entries in use
residual vector quantisation · K = 8 entries per codebook · 3 bits per stageresidual r2 after 2 stageszoom ×9.6 · rms 0.10● held-out vectors · ● codebook 3 (k-means on r2)02040608012345678SNR (dB) vs codebooks N_q— held-out··· train- - 9.03 dB/stageN_q = 2: held-out 19.6 dB, train 20.0 dBbitrate · EnCodec 24 kHz (1.5 / 3 / 6 / 12 / 24 kbps)24,000 / 320 = 75 frames/s × 8 codebooks × log₂1024 = 10 bits → 6.0 kbps600 tokens/s · 6,000 tokens for 10 s · 64× smaller than 16-bit PCMone flat VQ with the same rate would need 2^80 entries; RVQ stores 8 × 1,024 = 8,192 vectorstoy panel: 512 training and 512 held-out 2-D vectors; rule of thumb 6.02 dB per bit per dimension
codebook K
codec
fig.RVQ trained by k-means stage by stage on 512 toy 2-D frames and scored on 512 held-out ones. Default (K = 8, 2 stages): held-out SNR 19.6 dB, close to the 6.02 dB-per-bit rule (2 × 9.03 = 18.1 dB). Each extra codebook adds less: with 8 stages the held-out SNR stalls at 38.4 dB while training SNR reaches 58.5 dB, because late codebooks fit noise. K = 32 gives 17.6 dB from one stage. The calculator reproduces 6 kbps = 600 tokens/s for EnCodec 24 kHz with 8 codebooks (64× below 16-bit PCM), 2.2 kbps for MusicGen's codec and 7.75 kbps for DAC.

Worked example: ten seconds through EnCodec 24 kHz

BandwidthCodebooksTokens per sTokens in 10 sBits in 10 svs PCM
1.5 kbps21501,50015,000256×
3 kbps43003,00030,000128×
6 kbps86006,00060,00064×
12 kbps161,20012,000120,00032×
24 kbps322,40024,000240,00016×
240,000 samples → 750 frames. Bits for 10 s and compression against 16-bit PCM at 384 kbps

At 6 kbps the 10-second clip is 7,500 bytes, against 480,000 bytes of PCM. For a language model it is also 6,000 tokens, far more than the words spoken in it. That length is why audio LMs split the codebooks into coarse and fine stages or predict several codebooks per step.

python
1import numpy as np
2from transformers import AutoProcessor, EncodecModel
3
4model = EncodecModel.from_pretrained("facebook/encodec_24khz")
5processor = AutoProcessor.from_pretrained("facebook/encodec_24khz")
6
7sr = processor.sampling_rate # 24000
8t = np.arange(10 * sr) / sr
9audio = 0.3 * np.sin(2 * np.pi * 220 * t) * (1 + np.sin(2 * np.pi * 2 * t)) / 2
10
11inputs = processor(raw_audio=audio, sampling_rate=sr, return_tensors="pt")
12enc = model.encode(inputs["input_values"], inputs["padding_mask"], bandwidth=6.0)
13print(enc.audio_codes.shape) # (1, 1, 8, 750): chunks, batch, codebooks, frames
14print(enc.audio_codes.max() < 1024) # every token is a 10-bit index
15
16recon = model.decode(enc.audio_codes, enc.audio_scales, inputs["padding_mask"])[0]
17bits = enc.audio_codes.numel() * np.log2(1024)
18print(bits / 10, "bit/s") # 6000.0
Encode and decode with EnCodec through Hugging Face transformers (not run here: needs torch and a model download)

Check yourself

Why do SoundStream and EnCodec use residual VQ instead of one large codebook?

Check yourself

What does quantizer dropout buy?

Remember

  • ✓Conv encoder with strides 2·4·5·8 = 320: 24 kHz audio → 75 latent frames per second.
  • ✓RVQ: each of NqN_q codebooks quantises the residual of the previous ones; z^=∑iqi\hat z=\sum_i q_i.
  • ✓Bitrate R=(fs/M) Nqlog⁡2KR=(f_s/M)\,N_q\log_2K: EnCodec 24 kHz gives 1.5–24 kbps with 2–32 codebooks of 1024.
  • ✓Training: waveform + multi-scale mel reconstruction, adversarial and feature-matching losses, commitment loss.
  • ✓Quantizer dropout trains one model for every bitrate; the first codebooks carry the coarse content.
  • ✓The codes are the vocabulary of audio LMs: AudioLM (SoundStream), VALL-E (EnCodec 8 codebooks), MusicGen (EnCodec 32 kHz, 4 codebooks).
← back to Deep Learning roadmap