Autoregressive generative models

An autoregressive model factorises a joint distribution by the chain rule, p(x) = Πᵢ p(xᵢ | x<ᵢ), and learns every conditional with one shared network. The likelihood is exact and training is parallel under teacher forcing, but sampling needs one network pass per dimension.

14 min read1177 wordscore0 viewsupdated 2026-10-12

Definition

Autoregressive generative model A density model that fixes an ordering x1,…,xDx_1,\dots,x_D of the dimensions of xx and writes the joint distribution exactly as a product of one-dimensional conditionals, pθ(x)=∏i=1Dpθ(xi∣x<i)p_\theta(x)=\prod_{i=1}^{D}p_\theta(x_i\mid x_{<i}). A single network, made causal by masking so that output ii sees only inputs <i<i, produces all DD conditionals. Training maximises the exact log-likelihood. Sampling draws x1x_1, then x2x_2 given x1x_1, and so on (ancestral sampling).

The chain rule of probability holds for any joint distribution and any ordering, so the factorisation itself makes no assumption. All modelling choices sit in how each conditional is parameterised. Early models used a separate logistic regression per dimension (fully visible sigmoid belief networks, Frey 1998). Bengio et al. (2003) used a neural network for next-word prediction. Larochelle and Murray's NADE (AISTATS 2011) shared weights across the conditionals of binary images, and Germain, Gregor, Murray and Larochelle's MADE (ICML 2015) obtained all conditionals from one masked autoencoder pass. In 2016 van den Oord and colleagues applied the idea to raw pixels (PixelRNN and PixelCNN, ICML 2016) and to raw audio (WaveNet). The decoder-only language models from GPT (Radford et al. 2018) to GPT-3 (Brown et al., NeurIPS 2020) are the same construction over text tokens.

The objective: exact likelihood

log⁡pθ(x)=∑i=1Dlog⁡pθ(xi∣x<i),L(θ)=− Ex∼pdata[∑i=1Dlog⁡pθ(xi∣x<i)]=H(pdata)+KL(pdata ∥ pθ)\log p_\theta(x)=\sum_{i=1}^{D}\log p_\theta(x_i\mid x_{<i}),\qquad \mathcal L(\theta)=-\,\mathbb E_{x\sim p_{\text{data}}}\Big[\sum_{i=1}^{D}\log p_\theta(x_i\mid x_{<i})\Big]=H(p_{\text{data}})+\mathrm{KL}(p_{\text{data}}\,\|\,p_\theta)
x<ix_{<i}
the prefix (x1,…,xi−1)(x_1,\dots,x_{i-1}); x<1x_{<1} is empty, so the first factor is a marginal
pθ(xi∣x<i)p_\theta(x_i\mid x_{<i})
a categorical (softmax over 256 pixel levels or a token vocabulary), a Bernoulli, or a continuous density such as a mixture of logistics
L\mathcal L
the expected negative log-likelihood, a cross-entropy; it equals the data entropy plus the KL gap, so minimising it minimises KL(pdata∥pθ)\mathrm{KL}(p_{\text{data}}\|p_\theta)

No bound and no sampling are involved: the training loss is the exact NLL. This is unlike the Variational autoencoder (VAE), which optimises a lower bound, and the Generative adversarial network (GAN), which has no likelihood at all. Results on images are reported in bits per dimension, the NLL in base 2 divided by the number of dimensions:

bits/dim=−log⁡2pθ(x)D=LnatsDln⁡2\text{bits/dim}=\frac{-\log_2 p_\theta(x)}{D}=\frac{\mathcal L_{\text{nats}}}{D\ln 2}
DD
number of dimensions; 3⋅32⋅32=30723\cdot32\cdot32=3072 for a CIFAR-10 image
88
a uniform model over 256 levels scores exactly 8 bits/dim; PixelCNN-family models reach about 3 bits/dim on CIFAR-10

Teacher forcing and the cost of sampling

During training the whole ground-truth xx is known, so every conditional can be evaluated at once. The network receives the true prefix at each position (teacher forcing, Williams and Zipser 1989), and a causal mask stops position ii from reading xix_i or anything after it. One forward pass yields all DD log-probabilities. Masked convolutions (PixelCNN, WaveNet), masked dense layers (MADE) and causal self-attention (see Decoder-only transformer) all implement the same constraint.

Sampling cannot be parallelised this way. xix_i must be drawn before p(xi+1∣x≤i)p(x_{i+1}\mid x_{\le i}) can be computed, so a sample costs DD sequential network evaluations. A 16 kHz WaveNet needs 16,000 passes per second of audio. A 256×256256\times256 RGB image modelled pixel by pixel needs 196,608. Caching (the KV cache of transformers, the queue-based fast WaveNet) removes repeated work, but not the sequential dependence. This is why image models moved to short sequences of learned tokens (see Vector-quantized VAE (VQ-VAE) and Autoregressive image generation).

Making one network causal: NADE and MADE

NADE computes the hidden state for conditional ii as hi=σ(W:,<ix<i+c)h_i=\sigma(W_{:,<i}x_{<i}+c), sharing WW across ii, and updates it in O(H)O(H) per step, giving O(DH)O(DH) for all conditionals. MADE instead takes an ordinary autoencoder and multiplies each weight matrix by a binary mask. Every unit kk gets a degree m(k)m(k): inputs are numbered 1,…,D1,\dots,D, hidden degrees are drawn from {1,…,D−1}\{1,\dots,D-1\}, and connections are kept only where they cannot create a path from xjx_j to output ii with j≥ij\ge i:

Mk,dW=1[m(k)≥d],Md,kV=1[d>m(k)],x^=σ((V⊙MV) h(x)),  h(x)=g((W⊙MW) x+b)M^{W}_{k,d}=\mathbb 1\big[m(k)\ge d\big],\qquad M^{V}_{d,k}=\mathbb 1\big[d>m(k)\big],\qquad \hat x=\sigma\big((V\odot M^{V})\,h(x)\big),\ \ h(x)=g\big((W\odot M^{W})\,x+b\big)
m(k)m(k)
the largest input index unit kk may depend on
MWM^{W}
input-to-hidden mask; hidden-to-hidden masks use 1[m(k′)≥m(k)]\mathbb 1[m(k')\ge m(k)]
MVM^{V}
hidden-to-output mask; the strict inequality makes output dd depend only on x<dx_{<d}
x^d\hat x_d
p(xd=1∣x<d)p(x_d=1\mid x_{<d}); all DD of them come from one pass

The Jacobian ∂x^/∂x\partial\hat x/\partial x of a MADE is strictly lower-triangular, which is the autoregressive property. MADE also trains over many orderings by resampling the input order and the degrees. The same masked layers later became the building block of autoregressive flows (MAF, IAF; see Normalizing flows).

WaveNet and PixelCNN: causal convolutions

WaveNet (van den Oord et al., 2016) models raw audio at 16 kHz. Samples are μ\mu-law companded (μ=255\mu=255) to 256 levels and predicted with a 256-way softmax. The network is a stack of dilated causal convolutions with filter size 2. Dilations double from 1 to 512 and the block is repeated, so the receptive field grows exponentially with depth while the cost per layer stays constant. Each layer uses the gated unit tanh⁡(Wf∗x)⊙σ(Wg∗x)\tanh(W_f*x)\odot\sigma(W_g*x) with residual and skip connections.

receptive field=1+∑l(k−1) dl=1+(1+2+4+⋯+512)=1024 samples(k=2)\text{receptive field}=1+\sum_{l}(k-1)\,d_l=1+(1+2+4+\dots+512)=1024\ \text{samples}\quad(k=2)
kk
filter width (2 in WaveNet)
dld_l
dilation of layer ll; one block of 10 layers covers 1024 samples (64 ms at 16 kHz)

PixelCNN does the same on images in raster order, with k×kk\times k convolutions whose kernels are zeroed below and to the right of the centre. The first layer also hides the centre pixel (mask A), later layers keep it (mask B). The masks, the resulting blind spot and its fix are covered in Autoregressive image generation. Decoder-only transformers replace convolution by causal self-attention, which has an unbounded receptive field within the context window.

Landmark autoregressive models

NADE
Larochelle and Murray, AISTATS 2011; binary images (binarised MNIST), weights shared across conditionals, O(DH)O(DH) per evaluation
MADE
Germain et al., ICML 2015; masked autoencoder, all conditionals in one pass, order-agnostic training
PixelRNN / PixelCNN
van den Oord, Kalchbrenner and Kavukcuoglu, ICML 2016; 256-way softmax per sub-pixel; CIFAR-10 NLL 3.00 bits/dim (Diagonal BiLSTM), 3.14 (PixelCNN)
WaveNet
van den Oord et al., 2016; 16 kHz μ\mu-law audio; TTS mean opinion score 4.21 for US English, against 3.86 (concatenative) and 3.67 (parametric)
GPT-3
Brown et al., NeurIPS 2020; decoder-only transformer, 175B parameters, 2048-token context, next-token cross-entropy
q(x₁, x₂) = q(x₁) · q(x₂ | x₁) · shade = model joint, outline = true sliceq(x₁)x₁ → (24 bins on [−3, 3]) · ● 200 ancestral samples · ○ probe x*x₂↑q(x₂|x₁)data entropy H(p)5.067 natstest NLL = CE(p, q)5.339 natsKL(p ‖ q) = CE − H0.272 natsbits per dim3.851uniform baseline4.585 bits/dimtraining passes1, all positionssampling passes2 per sample (400)log q(x*) = log q(x₁ = 12) + log q(x₂ = 7 | x₁ = 12) = −2.97 + −2.90 = −5.86 natsa unimodal conditional cannot fit a slice with two bumps; the order now matters
data
conditional
order
fig.A 2-D density on [−3, 3]² discretised into 24 × 24 bins, so each sample is two tokens (x₁, x₂). A uniform model scores log₂ 24 = 4.585 bits/dim. The model is q(x₁)·q(x₂ | x₁), and every readout is an exact sum over the 576 bins. With categorical (24-way softmax) conditionals the model equals the data in either order: on two moons H = CE = 5.067 nats, KL = 0, 3.655 bits/dim. Swap in the best-fitting Gaussian conditional and every slice becomes one bump. Slices that cross both moons are then wrong: CE = 5.339 nats and KL = 0.272 in the order x₁ then x₂, but KL = 0.433 in the order x₂ then x₁, so with a limited model the ordering matters. Dropping the conditioning (q(x₁)q(x₂)) costs KL = 0.623. On 8 Gaussians the Gaussian conditional (KL 0.606) is worse than independence (0.508), because a slice through two blobs gets one wide bump between them. On the correlated Gaussian it is almost exact (KL 0.001), while independence costs 0.483. The red probe at x₁ bin 12 splits log q(x*) into its two chain-rule terms: −2.97 + −1.57 = −4.54 nats with categorical conditionals, −2.97 + −2.90 = −5.86 with the Gaussian one. Training evaluates both conditionals for all points in one pass. Each orange ancestral sample takes 2 sequential draws.

Worked example: three binary variables

Take D=3D=3 binary variables with logistic conditionals q(x1=1)=σ(0.4)q(x_1=1)=\sigma(0.4), q(x2=1∣x1)=σ(−0.5+1.5x1)q(x_2=1\mid x_1)=\sigma(-0.5+1.5x_1) and q(x3=1∣x1,x2)=σ(−1+x1+2x2)q(x_3=1\mid x_1,x_2)=\sigma(-1+x_1+2x_2). This is a fully visible sigmoid belief network, the simplest autoregressive model. The table lists the factor each conditional contributes for every outcome.

xq(x₁)q(x₂ | x₁)q(x₃ | x₁, x₂)q(x)−ln q(x) (nats)
(0, 0, 0)0.40130.62250.73110.18261.7004
(0, 0, 1)0.40130.62250.26890.06722.7004
(0, 1, 0)0.40130.37750.26890.04073.2004
(0, 1, 1)0.40130.37750.73110.11082.2004
(1, 0, 0)0.59870.26890.50000.08052.5194
(1, 0, 1)0.59870.26890.50000.08052.5194
(1, 1, 0)0.59870.73110.11920.05222.9532
(1, 1, 1)0.59870.73110.88080.38550.9532
All 8 outcomes: chain-rule factors, joint probability and NLL (computed in Python)

Scoring and sampling x = (1, 0, 1)

  1. 1Teacher forcing: feed the true prefix to every conditional at once. The logits are 0.40.4, −0.5+1.5=1.0-0.5+1.5=1.0 and −1+1+0=0-1+1+0=0, so q(x1=1)=0.5987q(x_1{=}1)=0.5987, q(x2=0∣x1=1)=1−σ(1)=0.2689q(x_2{=}0\mid x_1{=}1)=1-\sigma(1)=0.2689 and q(x3=1∣1,0)=σ(0)=0.5q(x_3{=}1\mid 1,0)=\sigma(0)=0.5.
  2. 2Exact likelihood: q(x)=0.5987⋅0.2689⋅0.5=0.0805q(x)=0.5987\cdot0.2689\cdot0.5=0.0805, so −ln⁡q(x)=0.5130+1.3133+0.6931=2.5194-\ln q(x)=0.5130+1.3133+0.6931=2.5194 nats, or 2.5194/(3ln⁡2)=1.21162.5194/(3\ln2)=1.2116 bits/dim. A uniform model scores 1 bit/dim on every outcome.
  3. 3The 8 probabilities sum to 1 by construction. No partition function is needed, because every factor is a normalised distribution.
  4. 4Sampling needs 3 sequential steps: draw x1x_1 from σ(0.4)\sigma(0.4), then compute σ(−0.5+1.5x1)\sigma(-0.5+1.5x_1) and draw x2x_2, then compute the third logit from both and draw x3x_3. The most likely outcome, (1,1,1)(1,1,1) with 0.38550.3855, is reached only if every draw goes that way.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class MaskedLinear(nn.Linear):
4 def __init__(self, n_in, n_out, mask):
5 super().__init__(n_in, n_out)
6 self.register_buffer("mask", mask.float()) # (n_out, n_in), 1 = connection allowed
7 def forward(self, x):
8 return F.linear(x, self.weight * self.mask, self.bias)
9
10class MADE(nn.Module):
11 """Binary MADE: one forward pass gives all D conditionals p(x_i = 1 | x_<i)."""
12 def __init__(self, D, H=128, seed=0):
13 super().__init__()
14 g = torch.Generator().manual_seed(seed)
15 m_in = torch.arange(1, D + 1) # input degrees 1..D (natural order)
16 m_h = torch.randint(1, D, (H,), generator=g) # hidden degrees in 1..D-1
17 M1 = m_h[:, None] >= m_in[None, :] # hidden k sees x_d iff m(k) >= d
18 M2 = m_h[:, None] >= m_h[None, :]
19 M3 = m_in[:, None] > m_h[None, :] # output d sees hidden k iff d > m(k)
20 self.net = nn.Sequential(MaskedLinear(D, H, M1), nn.ReLU(), MaskedLinear(H, H, M2), nn.ReLU(), MaskedLinear(H, D, M3))
21 self.D = D
22
23 def forward(self, x): # x (N, D) in {0, 1}
24 return self.net(x) # logits (N, D)
25
26 def nll(self, x): # exact −log p(x) in nats, teacher forcing
27 return F.binary_cross_entropy_with_logits(self(x), x, reduction="none").sum(-1)
28
29 @torch.no_grad()
30 def sample(self, n): # D sequential passes
31 x = torch.zeros(n, self.D)
32 for d in range(self.D):
33 x[:, d] = torch.bernoulli(torch.sigmoid(self(x)[:, d]))
34 return x
35
36torch.manual_seed(0)
37D = 16
38model = MADE(D)
39x = (torch.rand(256, D) < 0.3).float()
40opt = torch.optim.Adam(model.parameters(), lr=1e-3)
41for step in range(200):
42 loss = model.nll(x).mean()
43 opt.zero_grad(); loss.backward(); opt.step()
44print(model(x).shape, model.nll(x).shape) # torch.Size([256, 16]) torch.Size([256])
45
46# autoregressive check: output d must not depend on inputs d, d+1, ...
47J = torch.autograd.functional.jacobian(lambda v: model(v[None])[0], torch.rand(D))
48print(torch.equal(J.abs().gt(0), torch.tril(J.abs().gt(0), -1))) # True: Jacobian strictly lower-triangular
49
50# the conditionals multiply to a normalised distribution: sum p(x) over all 2^4 inputs of a D = 4 model
51small = MADE(4)
52allx = torch.tensor([[(i >> k) & 1 for k in range(4)] for i in range(16)]).float()
53print(torch.exp(-small.nll(allx)).sum().item()) # 1.0 (up to float error)
54s = model.sample(5)
55print(s.shape) # torch.Size([5, 16])
A binary MADE: masked layers, exact NLL with teacher forcing, a causality check and D-step sampling

Use when

  • +Exact, tractable log-likelihood; training is plain cross-entropy with no bound, no discriminator and no posterior
  • +Teacher forcing evaluates all conditionals in one parallel pass
  • +Universal: categorical conditionals can represent any discrete distribution
  • +Stable training that scales predictably with data and parameters (decoder-only language models)

Avoid when

  • −Sampling needs DD sequential passes; latency grows with sequence length
  • −A fixed ordering is arbitrary for images and other non-sequential data
  • −Exposure bias: trained on true prefixes, sampled on its own
  • −No low-dimensional latent code for representation learning or editing

Check yourself

Why can an autoregressive model be trained on a whole sequence in one forward pass but not sampled in one?

Check yourself

In a MADE with D = 4 inputs, a hidden unit has degree m(k) = 2. Which inputs can reach it and which outputs can it feed?

Remember

  • ✓Chain rule: pθ(x)=∏i=1Dpθ(xi∣x<i)p_\theta(x)=\prod_{i=1}^{D}p_\theta(x_i\mid x_{<i}) is exact for any ordering; training minimises the exact NLL =H(pdata)+KL(pdata∥pθ)=H(p_{\text{data}})+\mathrm{KL}(p_{\text{data}}\|p_\theta).
  • ✓Teacher forcing evaluates all DD conditionals in one masked pass; ancestral sampling needs DD sequential passes.
  • ✓Causality by masking: MADE masks 1[m(k)≥d]\mathbb 1[m(k)\ge d] and 1[d>m(k)]\mathbb 1[d>m(k)], masked convolutions (PixelCNN), dilated causal convolutions (WaveNet), causal self-attention (decoder-only LMs).
  • ✓Images are scored in bits/dim =Lnats/(Dln⁡2)=\mathcal L_{\text{nats}}/(D\ln2); uniform over 256 levels is 8, PixelCNN-family models about 3 on CIFAR-10.
  • ✓With limited conditionals the ordering matters; exposure bias and sequential sampling cost are the main weaknesses.
← back to Deep Learning roadmap