Music generation

Music is generated as symbolic note events (Music Transformer, relative attention) or as codec tokens of audio (Jukebox, MusicLM, MusicGen); MusicGen's delay pattern predicts 4 EnCodec codebooks in T + 3 steps instead of 4T.

8 min read784 wordsadvanced0 viewsupdated 2026-10-15

Definition

Music generation is sampling new music from a learned distribution, unconditionally or conditioned on text, a melody, lyrics or a musical prompt. Symbolic models generate notes (MIDI-like events), as in Music Transformer (Huang et al., 2018, ICLR 2019). Audio models generate the sound itself through codec tokens, as in Jukebox (Dhariwal et al., 2020), MusicLM (Agostinelli et al., 2023) and MusicGen (Copet et al., "Simple and Controllable Music Generation", NeurIPS 2023).

Music is long and structured. A three-minute song repeats motifs, returns to a chorus after a minute and keeps one key and tempo throughout. Symbolic data makes that structure short: a minute of piano is a few thousand events. It leaves out everything a performance adds, however: timbre, the voice, the mix. Audio models produce all of it, but a minute of audio is millions of samples, so they need the compression of Neural codec models and hierarchical or interleaved token schemes.

Symbolic: Music Transformer

Music Transformer encodes performances with a 388-token event vocabulary: 128 NOTE_ON and 128 NOTE_OFF pitches, 100 TIME_SHIFT values (10 ms to 1 s) and 32 velocity bins. A Decoder-only transformer models the event sequence. Its key change is relative attention: the logits get a term that depends on the distance between positions, so a pattern learned at one point in the piece applies anywhere, and pitch or timing intervals are easy to express.

RelAttn(Q,K,V)=softmax ⁣(QK⊤+SrelDh)V,Sijrel=qi⋅e j−i\mathrm{RelAttn}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top+S^{\text{rel}}}{\sqrt{D_h}}\Big)V,\qquad S^{\text{rel}}_{ij}=q_i\cdot e_{\,j-i}
ej−ie_{j-i}
learned embedding of the relative distance j−ij-i, shared across positions
SrelS^{\text{rel}}
computed as QEr⊤QE_r^\top followed by a "skewing" reshape, which cuts the extra memory from O(L2D)O(L^2D) to O(LD)O(LD)
DhD_h
head dimension

The memory saving made sequences of about 2,000 events practical, and on piano performances the model kept motifs and phrasing coherent over about a minute. Compare Rotary positional embeddings (RoPE), the relative scheme used by most LLMs today.

Audio: Jukebox, MusicLM, MusicGen

symbolic: notes as events (Music Transformer)piano roll, 1 barNOTE_ON 67VELOCITY 20TIME_SHIFT 250msNOTE_OFF 67NOTE_ON 64…388 events: 128 on + 128 off + 100 shifts + 32 velocitiesaudio: codec tokens of the waveformtimbre, voice, mix and productionare generated, not just the notes32 kHz → EnCodec: 50 frames/s × 4codebooks of 2048 = 200 tokens/s30 s = 6,000 tokensMusicGen (Copet et al. 2023): one Transformer, delay-interleaved codebooks"lo-fi beat, rain"T5 encoder→ cross-attentionoptional melodychroma: top pitch classdecoder-only Transformer300M / 1.5B / 3.3B · 4 embeddings summed in,4 softmax heads out, one step per columnc1c2c3c4codebook k shifted by k − 1 stepsEnCodecdecoder(frozen)→ 32 kHzmono audioT + 3 stepsfor T framesJukebox (Dhariwal et al. 2020)44.1 kHz raw audio, 3 VQ-VAE levelshop 8 / 32 / 128 → 5,512.5 / 1,378.1/ 344.5 tokens per secondtop prior writes, 2 upsamplers refine;artist, genre and lyrics conditioningMusicLM (Agostinelli et al. 2023)AudioLM hierarchy for music:MuLan text–music embedding →w2v-BERT semantic tokens →SoundStream acoustic tokens, 24 kHz+ MusicCaps eval set (5.5k clips)
fig.Top: a piano roll becomes NOTE_ON / TIME_SHIFT / NOTE_OFF events, while an audio model must produce the waveform (200 EnCodec tokens per second for MusicGen). Middle: MusicGen embeds the text with T5 for cross-attention, optionally adds a chromagram, and runs one Transformer over 4 codebooks offset by one step each, decoded by a frozen EnCodec. Bottom: Jukebox's three VQ-VAE levels and MusicLM's AudioLM-style hierarchy.

Jukebox works on 44.1 kHz raw audio. Three Vector-quantized VAE (VQ-VAE) levels compress it with hop lengths 8, 32 and 128 and 2,048-entry codebooks. A top-level Transformer prior generates the coarsest codes, conditioned on artist, genre and lyrics, and two upsampling priors fill in the finer levels. It can sing lyrics in a recognisable style, but sampling is very slow: hours of GPU time per minute of audio.

MusicLM reuses the AudioLM hierarchy (see Audio generation): w2v-BERT semantic tokens, then SoundStream acoustic tokens at 24 kHz. Text enters through MuLan, an embedding trained jointly on music and text. It can also follow a hummed or whistled melody. The authors released MusicCaps, 5.5k music clips with captions written by musicians, which became the standard text-to-music test set.

MusicGen removes the hierarchy. A single Decoder-only transformer predicts the 4 codebooks of a 32 kHz EnCodec (2,048 entries each, 50 frames per second). The codebook embeddings of a step are summed at the input, and 4 linear heads predict the 4 tokens at the output. Text is embedded with a T5 encoder and read through Cross-attention. For melody conditioning, a chromagram of a reference track, reduced to its dominant pitch class per frame, is given to the model as well.

The question is how to order KK parallel streams. Flattening puts all KTKT tokens in one sequence: exact, but KK times as many steps. Parallel predicts all codebooks of a frame at once, so a fine code cannot see the coarse code of its own frame. The delay pattern shifts codebook kk by k−1k-1 steps:

step(k,t)=t+k−1,p(c)≈∏s=1T+K−1  ∏k=1Kp(cs−k+1k ∣ all tokens of steps<s, text)\text{step}(k,t)=t+k-1,\qquad p(c)\approx\prod_{s=1}^{T+K-1}\;\prod_{k=1}^{K}p\big(c^{k}_{s-k+1}\,\big|\,\text{all tokens of steps}<s,\ \text{text}\big)
ctkc^k_t
token of codebook kk at frame tt (1-based)
T+K−1T+K-1
number of sequential steps: 1,503 for 30 s with K=4K=4, against 6,000 for flattening
≈\approx
within a step the KK tokens are sampled independently; but ctkc^k_t is predicted after ct1,…,ctk−1c^1_t,\dots,c^{k-1}_t, the dependency RVQ makes most important

Configurations and data

Music Transformer
388-event performance vocabulary; relative attention with the skewing trick; trained on Piano-e-Competition performances and J.S. Bach chorales
Jukebox
44.1 kHz; VQ-VAE hops 8 / 32 / 128 → 5,512.5 / 1,378.1 / 344.5 tokens per second; trained on 1.2 million songs with artist, genre and lyrics metadata
MusicLM
MuLan conditioning + w2v-BERT semantic tokens + SoundStream acoustic tokens, 24 kHz; released MusicCaps (5.5k captioned clips)
MusicGen
EnCodec 32 kHz, 4 × 2,048 codebooks at 50 Hz (2.2 kbps); delay pattern; 300M, 1.5B and 3.3B parameters; 20k hours of licensed music; evaluated on MusicCaps with FAD, KL and CLAP score
delay pattern · 4 codebooks × 50 frames/s · numbers = decoding stepclick a token · rows: RVQ codebooks (c1 coarsest) · columns: frames of 20 msc1c2c3c4123456789101112234567891011121334567891011121314456789101112131415123456789101112■ selected · ■ seen, earlier in exact order · ■ seen, a later frame · ■ dropped dependencyc4, frame 7: step 10 · sees 30 tokens · of the 27 earlier in the exact order it misses 0missed: 0 coarser same-frame, 0 earlier-frame · extra: 3 later-frame codesall 12 frames: dropped 0 same-frame + 105 past-frame deps · 43 later-frame codes seen30 s clip: 1,500 frames × 4 = 6,000 tokens in 1,503 sequential steps (flattening: 6,000)
pattern
fig.Every token is labelled with the step at which it is predicted, for 4 codebooks and 12 frames. Default (delay, c4 at frame 7): predicted at step 10, it sees all 27 tokens that precede it in the exact order plus 3 coarse codes of frames 8–9, and misses none; c1 at frame 7 misses 6 fine codes of frames 4–6. Over the grid, parallel drops 72 same-frame dependencies (6 per frame) and delay none. A 30 s clip takes 1,500 steps in parallel, 1,503 with delay and 6,000 flattened.

Worked example: 30 seconds of MusicGen

PatternSequential stepsSees coarser codes of own frame?Exact factorisation?
Flattening4 × 1,500 = 6,000yesyes
Parallel1,500nono
Delay1,500 + 3 = 1,503yesno: misses recent fine codes
Coarse first (VALL-E)1,500 + 3 = 1,503yes, all framesno: one codebook per parallel pass
50 frames/s × 30 s = 1,500 frames; 4 codebooks → 6,000 tokens in every pattern

For comparison, Jukebox's top level alone emits 344.53×60=20,672344.53\times60=20{,}672 tokens per minute of music and its bottom level 5,512.5×60=330,7505{,}512.5\times60=330{,}750. MusicGen covers the same minute with 3,0003{,}000 steps of 4 tokens each.

python
1import numpy as np
2
3def delay_pattern(codes, pad=-1):
4 """MusicGen's delay interleaving: codebook k is shifted right by k steps. codes: (K, T) -> (K, T + K - 1)."""
5 K, T = codes.shape
6 out = np.full((K, T + K - 1), pad, dtype=codes.dtype)
7 for k in range(K):
8 out[k, k:k + T] = codes[k]
9 return out
10
11def undelay(x):
12 K = x.shape[0]
13 T = x.shape[1] - K + 1
14 return np.stack([x[k, k:k + T] for k in range(K)])
15
16codes = np.random.default_rng(0).integers(0, 2048, size=(4, 1500)) # 30 s of EnCodec 32 kHz tokens
17x = delay_pattern(codes)
18print(x.shape) # (4, 1503): 1503 steps instead of 4 * 1500 = 6000
19print((undelay(x) == codes).all()) # True
20
21from transformers import AutoProcessor, MusicgenForConditionalGeneration
22processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
23model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
24inputs = processor(text=["lo-fi hip hop beat with mellow piano"], padding=True, return_tensors="pt")
25audio = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256) # 256 steps ≈ 5.12 s
26sr = model.config.audio_encoder.sampling_rate # 32000
Build and undo the delay pattern (numpy, runs as is), then sample MusicGen with Hugging Face transformers (second part not run here: needs torch and a download)

Check yourself

Why is MusicGen's delay pattern better than predicting all 4 codebooks of a frame in parallel, at almost the same cost?

Check yourself

How many sequential steps does a flattened model need for 10 s of MusicGen's codec, and how many with the delay pattern?

Remember

  • ✓Symbolic music: 388 MIDI-like events; Music Transformer adds relative attention with O(LD) memory via skewing.
  • ✓Jukebox: 44.1 kHz audio, 3 VQ-VAE levels (hops 8 / 32 / 128), a top prior plus 2 upsamplers; slow.
  • ✓MusicLM: AudioLM hierarchy (semantic → acoustic) with MuLan text conditioning; released MusicCaps.
  • ✓MusicGen: one Transformer over 4 EnCodec 32 kHz codebooks, T5 cross-attention, optional chroma melody.
  • ✓Delay pattern: token (k,t)(k,t) at step t+k−1t+k-1, so T+K−1T+K-1 steps instead of KTKT, and fine codes see their frame's coarse codes.
  • ✓Long-range form and training-data rights remain the open problems.
← back to Deep Learning roadmap