Definition
Neural audio codec is an encoder–quantiser–decoder network trained end to end to turn a waveform into a short stream of discrete codes and back. The two reference designs are SoundStream (Zeghidour, Luebs, Omran, Skoglund and Tagliasacchi, "SoundStream: An End-to-End Neural Audio Codec", 2021, IEEE/ACM TASLP) and EnCodec (Défossez, Copet, Synnaeve and Adi, "High Fidelity Neural Audio Compression", 2022). Both use a convolutional encoder, residual vector quantisation (RVQ) and a decoder trained with reconstruction and adversarial losses.
The task: take 24,000 samples per second of 16-bit audio (384 kbps) and send a few kilobits per second, so that the decoded audio still sounds like the original. Classic codecs such as Opus and EVS do this with hand-designed transforms and psychoacoustic models. At very low rates their output becomes muffled and buzzy. A neural codec learns the transform and the decoder from data, and its adversarial loss lets the decoder produce plausible detail that was never transmitted.
The discrete codes turned out to matter as much as the compression. A codec maps audio to a sequence of integers drawn from small vocabularies, so audio can be modelled by the same Transformers that model text. See Audio generation, Music generation and the codec language models in Text-to-speech.
Encoder, quantiser, decoder
At 24 kHz both encoders start with a 7-tap convolution and then apply four residual blocks with strides 2, 4, 5 and 8. The total stride is , so there are latent frames per second. The channels double at every downsampling. EnCodec adds a two-layer LSTM and a final convolution to dimensions. The decoder mirrors the encoder with transposed convolutions. SoundStream uses only causal convolutions, so it can stream with low latency.
A single vector quantiser cannot reach useful rates. At 6 kbps each frame gets bits, which would need a codebook of vectors. RVQ uses a cascade of small codebooks instead. The first stage quantises the frame, and every later stage quantises the error that is left:
- one encoder frame (-dimensional)
- entry of codebook , entries each in SoundStream and EnCodec at 24 kHz
- the transmitted index of stage , bits
- what the decoder receives; dropping the last stages gives a coarser
- latent frame rate: sample rate over total encoder stride
- number of codebooks actually sent
- bits per index
Codebooks are learned as in Vector-quantized VAE (VQ-VAE): the code vectors follow exponential moving averages of the frames assigned to them, a commitment loss pulls the encoder output towards the chosen codes, and gradients pass through the quantiser with the straight-through estimator. SoundStream initialises the codebooks with k-means on the first batch and replaces codes that go unused.
Losses and quantizer dropout
- mel spectrogram at scale ; EnCodec combines L1 and L2 over several window sizes
- generator term of the Adversarial loss; EnCodec uses a multi-scale STFT discriminator, SoundStream a waveform and an STFT discriminator
- L1 distance between discriminator features of and
- weight of the commitment term, summed over the RVQ stages
The reconstruction terms alone give dull, over-smoothed audio. The discriminators reward realistic fine structure, as in the GAN vocoders (HiFi-GAN). EnCodec also adds a loss balancer, which rescales each loss's gradient so that its weight sets its share of the total gradient, independent of its raw scale.
Quantizer dropout makes one model serve many bitrates. For each training example, SoundStream draws uniformly from and uses only the first stages. The decoder therefore learns to reconstruct from any prefix of the codes, and the sender picks the rate at inference by truncating. The paper reports that one such model matches models trained separately for each bitrate.
Published configurations
- SoundStream
- 24 kHz, 75 frames/s, 1024-entry codebooks; one model for 3–18 kbps with quantizer dropout; at 3 kbps it beat Opus at 12 kbps and approached EVS at 9.6 kbps in listening tests; runs in real time on a smartphone CPU
- EnCodec 24 kHz
- mono, 1.5 / 3 / 6 / 12 / 24 kbps = 2 / 4 / 8 / 16 / 32 codebooks of 1024 at 75 frames/s
- EnCodec 48 kHz
- stereo model at 3 / 6 / 12 / 24 kbps; an optional small Transformer entropy model compresses the codes further
- EnCodec 32 kHz
- the tokenizer of MusicGen: 4 codebooks of 2048 at 50 frames/s = 2.2 kbps
- DAC (Kumar et al., 2023)
- 44.1 kHz audio at about 8 kbps with 9 codebooks; codes looked up in a low-dimensional, L2-normalised space to keep all entries in use
Worked example: ten seconds through EnCodec 24 kHz
| Bandwidth | Codebooks | Tokens per s | Tokens in 10 s | Bits in 10 s | vs PCM |
|---|---|---|---|---|---|
| 1.5 kbps | 2 | 150 | 1,500 | 15,000 | 256× |
| 3 kbps | 4 | 300 | 3,000 | 30,000 | 128× |
| 6 kbps | 8 | 600 | 6,000 | 60,000 | 64× |
| 12 kbps | 16 | 1,200 | 12,000 | 120,000 | 32× |
| 24 kbps | 32 | 2,400 | 24,000 | 240,000 | 16× |
At 6 kbps the 10-second clip is 7,500 bytes, against 480,000 bytes of PCM. For a language model it is also 6,000 tokens, far more than the words spoken in it. That length is why audio LMs split the codebooks into coarse and fine stages or predict several codebooks per step.
1import numpy as np2from transformers import AutoProcessor, EncodecModel34model = EncodecModel.from_pretrained("facebook/encodec_24khz")5processor = AutoProcessor.from_pretrained("facebook/encodec_24khz")67sr = processor.sampling_rate # 240008t = np.arange(10 * sr) / sr9audio = 0.3 * np.sin(2 * np.pi * 220 * t) * (1 + np.sin(2 * np.pi * 2 * t)) / 21011inputs = processor(raw_audio=audio, sampling_rate=sr, return_tensors="pt")12enc = model.encode(inputs["input_values"], inputs["padding_mask"], bandwidth=6.0)13print(enc.audio_codes.shape) # (1, 1, 8, 750): chunks, batch, codebooks, frames14print(enc.audio_codes.max() < 1024) # every token is a 10-bit index1516recon = model.decode(enc.audio_codes, enc.audio_scales, inputs["padding_mask"])[0]17bits = enc.audio_codes.numel() * np.log2(1024)18print(bits / 10, "bit/s") # 6000.0Check yourself
Why do SoundStream and EnCodec use residual VQ instead of one large codebook?
Check yourself
What does quantizer dropout buy?
Remember
- ✓Conv encoder with strides 2·4·5·8 = 320: 24 kHz audio → 75 latent frames per second.
- ✓RVQ: each of codebooks quantises the residual of the previous ones; .
- ✓Bitrate : EnCodec 24 kHz gives 1.5–24 kbps with 2–32 codebooks of 1024.
- ✓Training: waveform + multi-scale mel reconstruction, adversarial and feature-matching losses, commitment loss.
- ✓Quantizer dropout trains one model for every bitrate; the first codebooks carry the coarse content.
- ✓The codes are the vocabulary of audio LMs: AudioLM (SoundStream), VALL-E (EnCodec 8 codebooks), MusicGen (EnCodec 32 kHz, 4 codebooks).