Definition
Autoregressive generative model A density model that fixes an ordering of the dimensions of and writes the joint distribution exactly as a product of one-dimensional conditionals, . A single network, made causal by masking so that output sees only inputs , produces all conditionals. Training maximises the exact log-likelihood. Sampling draws , then given , and so on (ancestral sampling).
The chain rule of probability holds for any joint distribution and any ordering, so the factorisation itself makes no assumption. All modelling choices sit in how each conditional is parameterised. Early models used a separate logistic regression per dimension (fully visible sigmoid belief networks, Frey 1998). Bengio et al. (2003) used a neural network for next-word prediction. Larochelle and Murray's NADE (AISTATS 2011) shared weights across the conditionals of binary images, and Germain, Gregor, Murray and Larochelle's MADE (ICML 2015) obtained all conditionals from one masked autoencoder pass. In 2016 van den Oord and colleagues applied the idea to raw pixels (PixelRNN and PixelCNN, ICML 2016) and to raw audio (WaveNet). The decoder-only language models from GPT (Radford et al. 2018) to GPT-3 (Brown et al., NeurIPS 2020) are the same construction over text tokens.
The objective: exact likelihood
- the prefix ; is empty, so the first factor is a marginal
- a categorical (softmax over 256 pixel levels or a token vocabulary), a Bernoulli, or a continuous density such as a mixture of logistics
- the expected negative log-likelihood, a cross-entropy; it equals the data entropy plus the KL gap, so minimising it minimises
No bound and no sampling are involved: the training loss is the exact NLL. This is unlike the Variational autoencoder (VAE), which optimises a lower bound, and the Generative adversarial network (GAN), which has no likelihood at all. Results on images are reported in bits per dimension, the NLL in base 2 divided by the number of dimensions:
- number of dimensions; for a CIFAR-10 image
- a uniform model over 256 levels scores exactly 8 bits/dim; PixelCNN-family models reach about 3 bits/dim on CIFAR-10
Teacher forcing and the cost of sampling
During training the whole ground-truth is known, so every conditional can be evaluated at once. The network receives the true prefix at each position (teacher forcing, Williams and Zipser 1989), and a causal mask stops position from reading or anything after it. One forward pass yields all log-probabilities. Masked convolutions (PixelCNN, WaveNet), masked dense layers (MADE) and causal self-attention (see Decoder-only transformer) all implement the same constraint.
Sampling cannot be parallelised this way. must be drawn before can be computed, so a sample costs sequential network evaluations. A 16 kHz WaveNet needs 16,000 passes per second of audio. A RGB image modelled pixel by pixel needs 196,608. Caching (the KV cache of transformers, the queue-based fast WaveNet) removes repeated work, but not the sequential dependence. This is why image models moved to short sequences of learned tokens (see Vector-quantized VAE (VQ-VAE) and Autoregressive image generation).
Making one network causal: NADE and MADE
NADE computes the hidden state for conditional as , sharing across , and updates it in per step, giving for all conditionals. MADE instead takes an ordinary autoencoder and multiplies each weight matrix by a binary mask. Every unit gets a degree : inputs are numbered , hidden degrees are drawn from , and connections are kept only where they cannot create a path from to output with :
- the largest input index unit may depend on
- input-to-hidden mask; hidden-to-hidden masks use
- hidden-to-output mask; the strict inequality makes output depend only on
- ; all of them come from one pass
The Jacobian of a MADE is strictly lower-triangular, which is the autoregressive property. MADE also trains over many orderings by resampling the input order and the degrees. The same masked layers later became the building block of autoregressive flows (MAF, IAF; see Normalizing flows).
WaveNet and PixelCNN: causal convolutions
WaveNet (van den Oord et al., 2016) models raw audio at 16 kHz. Samples are -law companded () to 256 levels and predicted with a 256-way softmax. The network is a stack of dilated causal convolutions with filter size 2. Dilations double from 1 to 512 and the block is repeated, so the receptive field grows exponentially with depth while the cost per layer stays constant. Each layer uses the gated unit with residual and skip connections.
- filter width (2 in WaveNet)
- dilation of layer ; one block of 10 layers covers 1024 samples (64 ms at 16 kHz)
PixelCNN does the same on images in raster order, with convolutions whose kernels are zeroed below and to the right of the centre. The first layer also hides the centre pixel (mask A), later layers keep it (mask B). The masks, the resulting blind spot and its fix are covered in Autoregressive image generation. Decoder-only transformers replace convolution by causal self-attention, which has an unbounded receptive field within the context window.
Landmark autoregressive models
- NADE
- Larochelle and Murray, AISTATS 2011; binary images (binarised MNIST), weights shared across conditionals, per evaluation
- MADE
- Germain et al., ICML 2015; masked autoencoder, all conditionals in one pass, order-agnostic training
- PixelRNN / PixelCNN
- van den Oord, Kalchbrenner and Kavukcuoglu, ICML 2016; 256-way softmax per sub-pixel; CIFAR-10 NLL 3.00 bits/dim (Diagonal BiLSTM), 3.14 (PixelCNN)
- WaveNet
- van den Oord et al., 2016; 16 kHz -law audio; TTS mean opinion score 4.21 for US English, against 3.86 (concatenative) and 3.67 (parametric)
- GPT-3
- Brown et al., NeurIPS 2020; decoder-only transformer, 175B parameters, 2048-token context, next-token cross-entropy
Worked example: three binary variables
Take binary variables with logistic conditionals , and . This is a fully visible sigmoid belief network, the simplest autoregressive model. The table lists the factor each conditional contributes for every outcome.
| x | q(x₁) | q(x₂ | x₁) | q(x₃ | x₁, x₂) | q(x) | −ln q(x) (nats) |
|---|---|---|---|---|---|
| (0, 0, 0) | 0.4013 | 0.6225 | 0.7311 | 0.1826 | 1.7004 |
| (0, 0, 1) | 0.4013 | 0.6225 | 0.2689 | 0.0672 | 2.7004 |
| (0, 1, 0) | 0.4013 | 0.3775 | 0.2689 | 0.0407 | 3.2004 |
| (0, 1, 1) | 0.4013 | 0.3775 | 0.7311 | 0.1108 | 2.2004 |
| (1, 0, 0) | 0.5987 | 0.2689 | 0.5000 | 0.0805 | 2.5194 |
| (1, 0, 1) | 0.5987 | 0.2689 | 0.5000 | 0.0805 | 2.5194 |
| (1, 1, 0) | 0.5987 | 0.7311 | 0.1192 | 0.0522 | 2.9532 |
| (1, 1, 1) | 0.5987 | 0.7311 | 0.8808 | 0.3855 | 0.9532 |
Scoring and sampling x = (1, 0, 1)
- 1Teacher forcing: feed the true prefix to every conditional at once. The logits are , and , so , and .
- 2Exact likelihood: , so nats, or bits/dim. A uniform model scores 1 bit/dim on every outcome.
- 3The 8 probabilities sum to 1 by construction. No partition function is needed, because every factor is a normalised distribution.
- 4Sampling needs 3 sequential steps: draw from , then compute and draw , then compute the third logit from both and draw . The most likely outcome, with , is reached only if every draw goes that way.
1import torch, torch.nn as nn, torch.nn.functional as F23class MaskedLinear(nn.Linear):4 def __init__(self, n_in, n_out, mask):5 super().__init__(n_in, n_out)6 self.register_buffer("mask", mask.float()) # (n_out, n_in), 1 = connection allowed7 def forward(self, x):8 return F.linear(x, self.weight * self.mask, self.bias)910class MADE(nn.Module):11 """Binary MADE: one forward pass gives all D conditionals p(x_i = 1 | x_<i)."""12 def __init__(self, D, H=128, seed=0):13 super().__init__()14 g = torch.Generator().manual_seed(seed)15 m_in = torch.arange(1, D + 1) # input degrees 1..D (natural order)16 m_h = torch.randint(1, D, (H,), generator=g) # hidden degrees in 1..D-117 M1 = m_h[:, None] >= m_in[None, :] # hidden k sees x_d iff m(k) >= d18 M2 = m_h[:, None] >= m_h[None, :]19 M3 = m_in[:, None] > m_h[None, :] # output d sees hidden k iff d > m(k)20 self.net = nn.Sequential(MaskedLinear(D, H, M1), nn.ReLU(), MaskedLinear(H, H, M2), nn.ReLU(), MaskedLinear(H, D, M3))21 self.D = D2223 def forward(self, x): # x (N, D) in {0, 1}24 return self.net(x) # logits (N, D)2526 def nll(self, x): # exact −log p(x) in nats, teacher forcing27 return F.binary_cross_entropy_with_logits(self(x), x, reduction="none").sum(-1)2829 @torch.no_grad()30 def sample(self, n): # D sequential passes31 x = torch.zeros(n, self.D)32 for d in range(self.D):33 x[:, d] = torch.bernoulli(torch.sigmoid(self(x)[:, d]))34 return x3536torch.manual_seed(0)37D = 1638model = MADE(D)39x = (torch.rand(256, D) < 0.3).float()40opt = torch.optim.Adam(model.parameters(), lr=1e-3)41for step in range(200):42 loss = model.nll(x).mean()43 opt.zero_grad(); loss.backward(); opt.step()44print(model(x).shape, model.nll(x).shape) # torch.Size([256, 16]) torch.Size([256])4546# autoregressive check: output d must not depend on inputs d, d+1, ...47J = torch.autograd.functional.jacobian(lambda v: model(v[None])[0], torch.rand(D))48print(torch.equal(J.abs().gt(0), torch.tril(J.abs().gt(0), -1))) # True: Jacobian strictly lower-triangular4950# the conditionals multiply to a normalised distribution: sum p(x) over all 2^4 inputs of a D = 4 model51small = MADE(4)52allx = torch.tensor([[(i >> k) & 1 for k in range(4)] for i in range(16)]).float()53print(torch.exp(-small.nll(allx)).sum().item()) # 1.0 (up to float error)54s = model.sample(5)55print(s.shape) # torch.Size([5, 16])Use when
- +Exact, tractable log-likelihood; training is plain cross-entropy with no bound, no discriminator and no posterior
- +Teacher forcing evaluates all conditionals in one parallel pass
- +Universal: categorical conditionals can represent any discrete distribution
- +Stable training that scales predictably with data and parameters (decoder-only language models)
Avoid when
- −Sampling needs sequential passes; latency grows with sequence length
- −A fixed ordering is arbitrary for images and other non-sequential data
- −Exposure bias: trained on true prefixes, sampled on its own
- −No low-dimensional latent code for representation learning or editing
Check yourself
Why can an autoregressive model be trained on a whole sequence in one forward pass but not sampled in one?
Check yourself
In a MADE with D = 4 inputs, a hidden unit has degree m(k) = 2. Which inputs can reach it and which outputs can it feed?
Remember
- ✓Chain rule: is exact for any ordering; training minimises the exact NLL .
- ✓Teacher forcing evaluates all conditionals in one masked pass; ancestral sampling needs sequential passes.
- ✓Causality by masking: MADE masks and , masked convolutions (PixelCNN), dilated causal convolutions (WaveNet), causal self-attention (decoder-only LMs).
- ✓Images are scored in bits/dim ; uniform over 256 levels is 8, PixelCNN-family models about 3 on CIFAR-10.
- ✓With limited conditionals the ordering matters; exposure bias and sequential sampling cost are the main weaknesses.