# Music generation

> Music is generated as symbolic note events (Music Transformer, relative attention) or as codec tokens of audio (Jukebox, MusicLM, MusicGen); MusicGen's delay pattern predicts 4 EnCodec codebooks in T + 3 steps instead of 4T.

Source: https://www.mlfoundry.in/learn/music-generation  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 8 min read · updated 2026-10-15

**Music generation:** is sampling new music from a learned distribution, unconditionally or conditioned on text, a melody, lyrics or a musical prompt. **Symbolic** models generate notes (MIDI-like events), as in Music Transformer (Huang et al., 2018, ICLR 2019). **Audio** models generate the sound itself through codec tokens, as in Jukebox (Dhariwal et al., 2020), MusicLM (Agostinelli et al., 2023) and MusicGen (Copet et al., "Simple and Controllable Music Generation", NeurIPS 2023).

Music is long and structured. A three-minute song repeats motifs, returns to a chorus after a minute and keeps one key and tempo throughout. Symbolic data makes that structure short: a minute of piano is a few thousand events. It leaves out everything a performance adds, however: timbre, the voice, the mix. Audio models produce all of it, but a minute of audio is millions of samples, so they need the compression of **Neural codec models** and hierarchical or interleaved token schemes.

## Symbolic: Music Transformer

Music Transformer encodes performances with a 388-token event vocabulary: 128 NOTE_ON and 128 NOTE_OFF pitches, 100 TIME_SHIFT values (10 ms to 1 s) and 32 velocity bins. A **Decoder-only transformer** models the event sequence. Its key change is **relative attention**: the logits get a term that depends on the distance between positions, so a pattern learned at one point in the piece applies anywhere, and pitch or timing intervals are easy to express.

$$
\mathrm{RelAttn}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top+S^{\text{rel}}}{\sqrt{D_h}}\Big)V,\qquad S^{\text{rel}}_{ij}=q_i\cdot e_{\,j-i}
$$

- $e_{j-i}$: learned embedding of the relative distance $j-i$, shared across positions
- $S^{\text{rel}}$: computed as $QE_r^\top$ followed by a "skewing" reshape, which cuts the extra memory from $O(L^2D)$ to $O(LD)$
- $D_h$: head dimension

The memory saving made sequences of about 2,000 events practical, and on piano performances the model kept motifs and phrasing coherent over about a minute. Compare **Rotary positional embeddings (RoPE)**, the relative scheme used by most LLMs today.

## Audio: Jukebox, MusicLM, MusicGen

> Figure: Top: a piano roll becomes NOTE_ON / TIME_SHIFT / NOTE_OFF events, while an audio model must produce the waveform (200 EnCodec tokens per second for MusicGen). Middle: MusicGen embeds the text with T5 for cross-attention, optionally adds a chromagram, and runs one Transformer over 4 codebooks offset by one step each, decoded by a frozen EnCodec. Bottom: Jukebox's three VQ-VAE levels and MusicLM's AudioLM-style hierarchy.

**Jukebox** works on 44.1 kHz raw audio. Three **Vector-quantized VAE (VQ-VAE)** levels compress it with hop lengths 8, 32 and 128 and 2,048-entry codebooks. A top-level Transformer prior generates the coarsest codes, conditioned on artist, genre and lyrics, and two upsampling priors fill in the finer levels. It can sing lyrics in a recognisable style, but sampling is very slow: hours of GPU time per minute of audio.

**MusicLM** reuses the AudioLM hierarchy (see **Audio generation**): w2v-BERT semantic tokens, then SoundStream acoustic tokens at 24 kHz. Text enters through MuLan, an embedding trained jointly on music and text. It can also follow a hummed or whistled melody. The authors released **MusicCaps**, 5.5k music clips with captions written by musicians, which became the standard text-to-music test set.

**MusicGen** removes the hierarchy. A single **Decoder-only transformer** predicts the 4 codebooks of a 32 kHz EnCodec (2,048 entries each, 50 frames per second). The codebook embeddings of a step are summed at the input, and 4 linear heads predict the 4 tokens at the output. Text is embedded with a T5 encoder and read through **Cross-attention**. For melody conditioning, a chromagram of a reference track, reduced to its dominant pitch class per frame, is given to the model as well.

The question is how to order $K$ parallel streams. **Flattening** puts all $KT$ tokens in one sequence: exact, but $K$ times as many steps. **Parallel** predicts all codebooks of a frame at once, so a fine code cannot see the coarse code of its own frame. The **delay** pattern shifts codebook $k$ by $k-1$ steps:

$$
\text{step}(k,t)=t+k-1,\qquad p(c)\approx\prod_{s=1}^{T+K-1}\;\prod_{k=1}^{K}p\big(c^{k}_{s-k+1}\,\big|\,\text{all tokens of steps}<s,\ \text{text}\big)
$$

- $c^k_t$: token of codebook $k$ at frame $t$ (1-based)
- $T+K-1$: number of sequential steps: 1,503 for 30 s with $K=4$, against 6,000 for flattening
- $\approx$: within a step the $K$ tokens are sampled independently; but $c^k_t$ is predicted after $c^1_t,\dots,c^{k-1}_t$, the dependency RVQ makes most important

**Configurations and data**

- **Music Transformer:** 388-event performance vocabulary; relative attention with the skewing trick; trained on Piano-e-Competition performances and J.S. Bach chorales
- **Jukebox:** 44.1 kHz; VQ-VAE hops 8 / 32 / 128 → 5,512.5 / 1,378.1 / 344.5 tokens per second; trained on 1.2 million songs with artist, genre and lyrics metadata
- **MusicLM:** MuLan conditioning + w2v-BERT semantic tokens + SoundStream acoustic tokens, 24 kHz; released MusicCaps (5.5k captioned clips)
- **MusicGen:** EnCodec 32 kHz, 4 × 2,048 codebooks at 50 Hz (2.2 kbps); delay pattern; 300M, 1.5B and 3.3B parameters; 20k hours of licensed music; evaluated on MusicCaps with FAD, KL and CLAP score

> Figure: Every token is labelled with the step at which it is predicted, for 4 codebooks and 12 frames. Default (delay, c4 at frame 7): predicted at step 10, it sees all 27 tokens that precede it in the exact order plus 3 coarse codes of frames 8–9, and misses none; c1 at frame 7 misses 6 fine codes of frames 4–6. Over the grid, parallel drops 72 same-frame dependencies (6 per frame) and delay none. A 30 s clip takes 1,500 steps in parallel, 1,503 with delay and 6,000 flattened.

## Worked example: 30 seconds of MusicGen

50 frames/s × 30 s = 1,500 frames; 4 codebooks → 6,000 tokens in every pattern

| Pattern | Sequential steps | Sees coarser codes of own frame? | Exact factorisation? |
| --- | --- | --- | --- |
| Flattening | 4 × 1,500 = 6,000 | yes | yes |
| Parallel | 1,500 | no | no |
| Delay | 1,500 + 3 = 1,503 | yes | no: misses recent fine codes |
| Coarse first (VALL-E) | 1,500 + 3 = 1,503 | yes, all frames | no: one codebook per parallel pass |

For comparison, Jukebox's top level alone emits $344.53\times60=20{,}672$ tokens per minute of music and its bottom level $5{,}512.5\times60=330{,}750$. MusicGen covers the same minute with $3{,}000$ steps of 4 tokens each.

```python
import numpy as np

def delay_pattern(codes, pad=-1):
    """MusicGen's delay interleaving: codebook k is shifted right by k steps. codes: (K, T) -> (K, T + K - 1)."""
    K, T = codes.shape
    out = np.full((K, T + K - 1), pad, dtype=codes.dtype)
    for k in range(K):
        out[k, k:k + T] = codes[k]
    return out

def undelay(x):
    K = x.shape[0]
    T = x.shape[1] - K + 1
    return np.stack([x[k, k:k + T] for k in range(K)])

codes = np.random.default_rng(0).integers(0, 2048, size=(4, 1500))   # 30 s of EnCodec 32 kHz tokens
x = delay_pattern(codes)
print(x.shape)                       # (4, 1503): 1503 steps instead of 4 * 1500 = 6000
print((undelay(x) == codes).all())   # True

from transformers import AutoProcessor, MusicgenForConditionalGeneration
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
inputs = processor(text=["lo-fi hip hop beat with mellow piano"], padding=True, return_tensors="pt")
audio = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)  # 256 steps ≈ 5.12 s
sr = model.config.audio_encoder.sampling_rate                                          # 32000
```

Build and undo the delay pattern (numpy, runs as is), then sample MusicGen with Hugging Face transformers (second part not run here: needs torch and a download)

> **Long-range structure is still the weak point:** Audio models are trained on fixed windows (MusicGen on 30-second segments), and a Transformer cannot repeat a chorus it can no longer see. Longer pieces are made by sliding the window and conditioning on the last seconds, which keeps continuity but rarely produces verse–chorus form. Symbolic models see far more of a piece per token, which is why structure was first shown there.

> **Data, copyright and memorisation:** Music is copyrighted, and recordings carry the rights of composers, performers and labels. Jukebox was trained on scraped songs with artist labels, which made style imitation of named artists a direct feature. MusicGen was trained only on licensed music. MusicLM's authors measured how often outputs matched training clips and did not release the model at publication. Open issues include consent for training data, verbatim regurgitation of melodies or lyrics, and imitation of a living artist's voice.

**Question:** Why is MusicGen's delay pattern better than predicting all 4 codebooks of a frame in parallel, at almost the same cost?

- A. It uses fewer tokens
- B. Each finer codebook of a frame is predicted one step after the coarser ones, so it conditions on them; parallel prediction samples them independently
- C. It makes the factorisation exact
- D. It removes the need for a codec decoder

**Answer:** B. Each finer codebook of a frame is predicted one step after the coarser ones, so it conditions on them; parallel prediction samples them independently — With delay, $c^k_t$ is predicted at step $t+k-1$, after $c^1_t,\dots,c^{k-1}_t$. Because RVQ codes each level from the residual of the previous ones, that is the dependency that matters most. It costs only $K-1=3$ extra steps. It is still not exact: $c^1_t$ is sampled before the fine codes of frames $t-1,\dots,t-3$.

**Question:** How many sequential steps does a flattened model need for 10 s of MusicGen's codec, and how many with the delay pattern?

- A. 500 and 500
- B. 2,000 and 503
- C. 2,000 and 2,003
- D. 4,000 and 1,003

**Answer:** B. 2,000 and 503 — $10\times50=500$ frames. Flattening: $4\times500=2{,}000$. Delay: $500+4-1=503$.

**Key points**

- Symbolic music: 388 MIDI-like events; Music Transformer adds relative attention with O(LD) memory via skewing.
- Jukebox: 44.1 kHz audio, 3 VQ-VAE levels (hops 8 / 32 / 128), a top prior plus 2 upsamplers; slow.
- MusicLM: AudioLM hierarchy (semantic → acoustic) with MuLan text conditioning; released MusicCaps.
- MusicGen: one Transformer over 4 EnCodec 32 kHz codebooks, T5 cross-attention, optional chroma melody.
- Delay pattern: token $(k,t)$ at step $t+k-1$, so $T+K-1$ steps instead of $KT$, and fine codes see their frame's coarse codes.
- Long-range form and training-data rights remain the open problems.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
