Definition
Music generation is sampling new music from a learned distribution, unconditionally or conditioned on text, a melody, lyrics or a musical prompt. Symbolic models generate notes (MIDI-like events), as in Music Transformer (Huang et al., 2018, ICLR 2019). Audio models generate the sound itself through codec tokens, as in Jukebox (Dhariwal et al., 2020), MusicLM (Agostinelli et al., 2023) and MusicGen (Copet et al., "Simple and Controllable Music Generation", NeurIPS 2023).
Music is long and structured. A three-minute song repeats motifs, returns to a chorus after a minute and keeps one key and tempo throughout. Symbolic data makes that structure short: a minute of piano is a few thousand events. It leaves out everything a performance adds, however: timbre, the voice, the mix. Audio models produce all of it, but a minute of audio is millions of samples, so they need the compression of Neural codec models and hierarchical or interleaved token schemes.
Symbolic: Music Transformer
Music Transformer encodes performances with a 388-token event vocabulary: 128 NOTE_ON and 128 NOTE_OFF pitches, 100 TIME_SHIFT values (10 ms to 1 s) and 32 velocity bins. A Decoder-only transformer models the event sequence. Its key change is relative attention: the logits get a term that depends on the distance between positions, so a pattern learned at one point in the piece applies anywhere, and pitch or timing intervals are easy to express.
- learned embedding of the relative distance , shared across positions
- computed as followed by a "skewing" reshape, which cuts the extra memory from to
- head dimension
The memory saving made sequences of about 2,000 events practical, and on piano performances the model kept motifs and phrasing coherent over about a minute. Compare Rotary positional embeddings (RoPE), the relative scheme used by most LLMs today.
Audio: Jukebox, MusicLM, MusicGen
Jukebox works on 44.1 kHz raw audio. Three Vector-quantized VAE (VQ-VAE) levels compress it with hop lengths 8, 32 and 128 and 2,048-entry codebooks. A top-level Transformer prior generates the coarsest codes, conditioned on artist, genre and lyrics, and two upsampling priors fill in the finer levels. It can sing lyrics in a recognisable style, but sampling is very slow: hours of GPU time per minute of audio.
MusicLM reuses the AudioLM hierarchy (see Audio generation): w2v-BERT semantic tokens, then SoundStream acoustic tokens at 24 kHz. Text enters through MuLan, an embedding trained jointly on music and text. It can also follow a hummed or whistled melody. The authors released MusicCaps, 5.5k music clips with captions written by musicians, which became the standard text-to-music test set.
MusicGen removes the hierarchy. A single Decoder-only transformer predicts the 4 codebooks of a 32 kHz EnCodec (2,048 entries each, 50 frames per second). The codebook embeddings of a step are summed at the input, and 4 linear heads predict the 4 tokens at the output. Text is embedded with a T5 encoder and read through Cross-attention. For melody conditioning, a chromagram of a reference track, reduced to its dominant pitch class per frame, is given to the model as well.
The question is how to order parallel streams. Flattening puts all tokens in one sequence: exact, but times as many steps. Parallel predicts all codebooks of a frame at once, so a fine code cannot see the coarse code of its own frame. The delay pattern shifts codebook by steps:
- token of codebook at frame (1-based)
- number of sequential steps: 1,503 for 30 s with , against 6,000 for flattening
- within a step the tokens are sampled independently; but is predicted after , the dependency RVQ makes most important
Configurations and data
- Music Transformer
- 388-event performance vocabulary; relative attention with the skewing trick; trained on Piano-e-Competition performances and J.S. Bach chorales
- Jukebox
- 44.1 kHz; VQ-VAE hops 8 / 32 / 128 → 5,512.5 / 1,378.1 / 344.5 tokens per second; trained on 1.2 million songs with artist, genre and lyrics metadata
- MusicLM
- MuLan conditioning + w2v-BERT semantic tokens + SoundStream acoustic tokens, 24 kHz; released MusicCaps (5.5k captioned clips)
- MusicGen
- EnCodec 32 kHz, 4 × 2,048 codebooks at 50 Hz (2.2 kbps); delay pattern; 300M, 1.5B and 3.3B parameters; 20k hours of licensed music; evaluated on MusicCaps with FAD, KL and CLAP score
Worked example: 30 seconds of MusicGen
| Pattern | Sequential steps | Sees coarser codes of own frame? | Exact factorisation? |
|---|---|---|---|
| Flattening | 4 × 1,500 = 6,000 | yes | yes |
| Parallel | 1,500 | no | no |
| Delay | 1,500 + 3 = 1,503 | yes | no: misses recent fine codes |
| Coarse first (VALL-E) | 1,500 + 3 = 1,503 | yes, all frames | no: one codebook per parallel pass |
For comparison, Jukebox's top level alone emits tokens per minute of music and its bottom level . MusicGen covers the same minute with steps of 4 tokens each.
1import numpy as np23def delay_pattern(codes, pad=-1):4 """MusicGen's delay interleaving: codebook k is shifted right by k steps. codes: (K, T) -> (K, T + K - 1)."""5 K, T = codes.shape6 out = np.full((K, T + K - 1), pad, dtype=codes.dtype)7 for k in range(K):8 out[k, k:k + T] = codes[k]9 return out1011def undelay(x):12 K = x.shape[0]13 T = x.shape[1] - K + 114 return np.stack([x[k, k:k + T] for k in range(K)])1516codes = np.random.default_rng(0).integers(0, 2048, size=(4, 1500)) # 30 s of EnCodec 32 kHz tokens17x = delay_pattern(codes)18print(x.shape) # (4, 1503): 1503 steps instead of 4 * 1500 = 600019print((undelay(x) == codes).all()) # True2021from transformers import AutoProcessor, MusicgenForConditionalGeneration22processor = AutoProcessor.from_pretrained("facebook/musicgen-small")23model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")24inputs = processor(text=["lo-fi hip hop beat with mellow piano"], padding=True, return_tensors="pt")25audio = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256) # 256 steps ≈ 5.12 s26sr = model.config.audio_encoder.sampling_rate # 32000Check yourself
Why is MusicGen's delay pattern better than predicting all 4 codebooks of a frame in parallel, at almost the same cost?
Check yourself
How many sequential steps does a flattened model need for 10 s of MusicGen's codec, and how many with the delay pattern?
Remember
- ✓Symbolic music: 388 MIDI-like events; Music Transformer adds relative attention with O(LD) memory via skewing.
- ✓Jukebox: 44.1 kHz audio, 3 VQ-VAE levels (hops 8 / 32 / 128), a top prior plus 2 upsamplers; slow.
- ✓MusicLM: AudioLM hierarchy (semantic → acoustic) with MuLan text conditioning; released MusicCaps.
- ✓MusicGen: one Transformer over 4 EnCodec 32 kHz codebooks, T5 cross-attention, optional chroma melody.
- ✓Delay pattern: token at step , so steps instead of , and fine codes see their frame's coarse codes.
- ✓Long-range form and training-data rights remain the open problems.