# Autoregressive generative models

> An autoregressive model factorises a joint distribution by the chain rule, p(x) = Πᵢ p(xᵢ | x<ᵢ), and learns every conditional with one shared network. The likelihood is exact and training is parallel under teacher forcing, but sampling needs one network pass per dimension.

Source: https://www.mlfoundry.in/learn/autoregressive-generative-models  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: core · 14 min read · updated 2026-10-12

**Autoregressive generative model:** A density model that fixes an ordering $x_1,\dots,x_D$ of the dimensions of $x$ and writes the joint distribution exactly as a product of one-dimensional conditionals, $p_\theta(x)=\prod_{i=1}^{D}p_\theta(x_i\mid x_{<i})$. A single network, made **causal** by masking so that output $i$ sees only inputs $<i$, produces all $D$ conditionals. Training maximises the exact log-likelihood. Sampling draws $x_1$, then $x_2$ given $x_1$, and so on (ancestral sampling).

The chain rule of probability holds for any joint distribution and any ordering, so the factorisation itself makes no assumption. All modelling choices sit in how each conditional is parameterised. Early models used a separate logistic regression per dimension (fully visible sigmoid belief networks, Frey 1998). Bengio et al. (2003) used a neural network for next-word prediction. Larochelle and Murray's **NADE** (AISTATS 2011) shared weights across the conditionals of binary images, and Germain, Gregor, Murray and Larochelle's **MADE** (ICML 2015) obtained all conditionals from one masked autoencoder pass. In 2016 van den Oord and colleagues applied the idea to raw pixels (PixelRNN and PixelCNN, ICML 2016) and to raw audio (WaveNet). The decoder-only language models from GPT (Radford et al. 2018) to GPT-3 (Brown et al., NeurIPS 2020) are the same construction over text tokens.

## The objective: exact likelihood

$$
\log p_\theta(x)=\sum_{i=1}^{D}\log p_\theta(x_i\mid x_{<i}),\qquad \mathcal L(\theta)=-\,\mathbb E_{x\sim p_{\text{data}}}\Big[\sum_{i=1}^{D}\log p_\theta(x_i\mid x_{<i})\Big]=H(p_{\text{data}})+\mathrm{KL}(p_{\text{data}}\,\|\,p_\theta)
$$

- $x_{<i}$: the prefix $(x_1,\dots,x_{i-1})$; $x_{<1}$ is empty, so the first factor is a marginal
- $p_\theta(x_i\mid x_{<i})$: a categorical (softmax over 256 pixel levels or a token vocabulary), a Bernoulli, or a continuous density such as a mixture of logistics
- $\mathcal L$: the expected negative log-likelihood, a cross-entropy; it equals the data entropy plus the KL gap, so minimising it minimises $\mathrm{KL}(p_{\text{data}}\|p_\theta)$

No bound and no sampling are involved: the training loss is the exact NLL. This is unlike the **Variational autoencoder (VAE)**, which optimises a lower bound, and the **Generative adversarial network (GAN)**, which has no likelihood at all. Results on images are reported in **bits per dimension**, the NLL in base 2 divided by the number of dimensions:

$$
\text{bits/dim}=\frac{-\log_2 p_\theta(x)}{D}=\frac{\mathcal L_{\text{nats}}}{D\ln 2}
$$

- $D$: number of dimensions; $3\cdot32\cdot32=3072$ for a CIFAR-10 image
- $8$: a uniform model over 256 levels scores exactly 8 bits/dim; PixelCNN-family models reach about 3 bits/dim on CIFAR-10

## Teacher forcing and the cost of sampling

During training the whole ground-truth $x$ is known, so every conditional can be evaluated at once. The network receives the true prefix at each position (**teacher forcing**, Williams and Zipser 1989), and a causal mask stops position $i$ from reading $x_i$ or anything after it. One forward pass yields all $D$ log-probabilities. Masked convolutions (PixelCNN, WaveNet), masked dense layers (MADE) and causal self-attention (see **Decoder-only transformer**) all implement the same constraint.

Sampling cannot be parallelised this way. $x_i$ must be drawn before $p(x_{i+1}\mid x_{\le i})$ can be computed, so a sample costs $D$ **sequential** network evaluations. A 16 kHz WaveNet needs 16,000 passes per second of audio. A $256\times256$ RGB image modelled pixel by pixel needs 196,608. Caching (the KV cache of transformers, the queue-based fast WaveNet) removes repeated work, but not the sequential dependence. This is why image models moved to short sequences of learned tokens (see **Vector-quantized VAE (VQ-VAE)** and **Autoregressive image generation**).

> **Exposure bias:** With teacher forcing the network only ever conditions on **real** prefixes. At sampling time it conditions on its **own** earlier draws, which may be slightly off-distribution, and errors can compound along the sequence. Scheduled sampling (Bengio et al. 2015) and sequence-level training (Ranzato et al. 2016) were proposed as fixes. Maximum likelihood with teacher forcing is still the default, because it is the exact objective of the factorised model.

## Making one network causal: NADE and MADE

NADE computes the hidden state for conditional $i$ as $h_i=\sigma(W_{:,<i}x_{<i}+c)$, sharing $W$ across $i$, and updates it in $O(H)$ per step, giving $O(DH)$ for all conditionals. MADE instead takes an ordinary autoencoder and multiplies each weight matrix by a binary mask. Every unit $k$ gets a **degree** $m(k)$: inputs are numbered $1,\dots,D$, hidden degrees are drawn from $\{1,\dots,D-1\}$, and connections are kept only where they cannot create a path from $x_j$ to output $i$ with $j\ge i$:

$$
M^{W}_{k,d}=\mathbb 1\big[m(k)\ge d\big],\qquad M^{V}_{d,k}=\mathbb 1\big[d>m(k)\big],\qquad \hat x=\sigma\big((V\odot M^{V})\,h(x)\big),\ \ h(x)=g\big((W\odot M^{W})\,x+b\big)
$$

- $m(k)$: the largest input index unit $k$ may depend on
- $M^{W}$: input-to-hidden mask; hidden-to-hidden masks use $\mathbb 1[m(k')\ge m(k)]$
- $M^{V}$: hidden-to-output mask; the strict inequality makes output $d$ depend only on $x_{<d}$
- $\hat x_d$: $p(x_d=1\mid x_{<d})$; all $D$ of them come from one pass

The Jacobian $\partial\hat x/\partial x$ of a MADE is strictly lower-triangular, which is the autoregressive property. MADE also trains over many orderings by resampling the input order and the degrees. The same masked layers later became the building block of autoregressive flows (MAF, IAF; see **Normalizing flows**).

## WaveNet and PixelCNN: causal convolutions

WaveNet (van den Oord et al., 2016) models raw audio at 16 kHz. Samples are $\mu$-law companded ($\mu=255$) to 256 levels and predicted with a 256-way softmax. The network is a stack of **dilated causal convolutions** with filter size 2. Dilations double from 1 to 512 and the block is repeated, so the receptive field grows exponentially with depth while the cost per layer stays constant. Each layer uses the gated unit $\tanh(W_f*x)\odot\sigma(W_g*x)$ with residual and skip connections.

$$
\text{receptive field}=1+\sum_{l}(k-1)\,d_l=1+(1+2+4+\dots+512)=1024\ \text{samples}\quad(k=2)
$$

- $k$: filter width (2 in WaveNet)
- $d_l$: dilation of layer $l$; one block of 10 layers covers 1024 samples (64 ms at 16 kHz)

PixelCNN does the same on images in raster order, with $k\times k$ convolutions whose kernels are zeroed below and to the right of the centre. The first layer also hides the centre pixel (mask A), later layers keep it (mask B). The masks, the resulting blind spot and its fix are covered in **Autoregressive image generation**. Decoder-only transformers replace convolution by causal self-attention, which has an unbounded receptive field within the context window.

**Landmark autoregressive models**

- **NADE:** Larochelle and Murray, AISTATS 2011; binary images (binarised MNIST), weights shared across conditionals, $O(DH)$ per evaluation
- **MADE:** Germain et al., ICML 2015; masked autoencoder, all conditionals in one pass, order-agnostic training
- **PixelRNN / PixelCNN:** van den Oord, Kalchbrenner and Kavukcuoglu, ICML 2016; 256-way softmax per sub-pixel; CIFAR-10 NLL 3.00 bits/dim (Diagonal BiLSTM), 3.14 (PixelCNN)
- **WaveNet:** van den Oord et al., 2016; 16 kHz $\mu$-law audio; TTS mean opinion score 4.21 for US English, against 3.86 (concatenative) and 3.67 (parametric)
- **GPT-3:** Brown et al., NeurIPS 2020; decoder-only transformer, 175B parameters, 2048-token context, next-token cross-entropy

> Figure: A 2-D density on [−3, 3]² discretised into 24 × 24 bins, so each sample is two tokens (x₁, x₂). A uniform model scores log₂ 24 = 4.585 bits/dim. The model is q(x₁)·q(x₂ | x₁), and every readout is an exact sum over the 576 bins. With categorical (24-way softmax) conditionals the model equals the data in either order: on two moons H = CE = 5.067 nats, KL = 0, 3.655 bits/dim. Swap in the best-fitting Gaussian conditional and every slice becomes one bump. Slices that cross both moons are then wrong: CE = 5.339 nats and KL = 0.272 in the order x₁ then x₂, but KL = 0.433 in the order x₂ then x₁, so with a limited model the ordering matters. Dropping the conditioning (q(x₁)q(x₂)) costs KL = 0.623. On 8 Gaussians the Gaussian conditional (KL 0.606) is worse than independence (0.508), because a slice through two blobs gets one wide bump between them. On the correlated Gaussian it is almost exact (KL 0.001), while independence costs 0.483. The red probe at x₁ bin 12 splits log q(x*) into its two chain-rule terms: −2.97 + −1.57 = −4.54 nats with categorical conditionals, −2.97 + −2.90 = −5.86 with the Gaussian one. Training evaluates both conditionals for all points in one pass. Each orange ancestral sample takes 2 sequential draws.

## Worked example: three binary variables

Take $D=3$ binary variables with logistic conditionals $q(x_1=1)=\sigma(0.4)$, $q(x_2=1\mid x_1)=\sigma(-0.5+1.5x_1)$ and $q(x_3=1\mid x_1,x_2)=\sigma(-1+x_1+2x_2)$. This is a fully visible sigmoid belief network, the simplest autoregressive model. The table lists the factor each conditional contributes for every outcome.

All 8 outcomes: chain-rule factors, joint probability and NLL (computed in Python)

| x | q(x₁) | q(x₂ \| x₁) | q(x₃ \| x₁, x₂) | q(x) | −ln q(x) (nats) |
| --- | --- | --- | --- | --- | --- |
| (0, 0, 0) | 0.4013 | 0.6225 | 0.7311 | 0.1826 | 1.7004 |
| (0, 0, 1) | 0.4013 | 0.6225 | 0.2689 | 0.0672 | 2.7004 |
| (0, 1, 0) | 0.4013 | 0.3775 | 0.2689 | 0.0407 | 3.2004 |
| (0, 1, 1) | 0.4013 | 0.3775 | 0.7311 | 0.1108 | 2.2004 |
| (1, 0, 0) | 0.5987 | 0.2689 | 0.5000 | 0.0805 | 2.5194 |
| (1, 0, 1) | 0.5987 | 0.2689 | 0.5000 | 0.0805 | 2.5194 |
| (1, 1, 0) | 0.5987 | 0.7311 | 0.1192 | 0.0522 | 2.9532 |
| (1, 1, 1) | 0.5987 | 0.7311 | 0.8808 | 0.3855 | 0.9532 |

**Scoring and sampling x = (1, 0, 1)**

1. Teacher forcing: feed the true prefix to every conditional at once. The logits are $0.4$, $-0.5+1.5=1.0$ and $-1+1+0=0$, so $q(x_1{=}1)=0.5987$, $q(x_2{=}0\mid x_1{=}1)=1-\sigma(1)=0.2689$ and $q(x_3{=}1\mid 1,0)=\sigma(0)=0.5$.
2. Exact likelihood: $q(x)=0.5987\cdot0.2689\cdot0.5=0.0805$, so $-\ln q(x)=0.5130+1.3133+0.6931=2.5194$ nats, or $2.5194/(3\ln2)=1.2116$ bits/dim. A uniform model scores 1 bit/dim on every outcome.
3. The 8 probabilities sum to 1 by construction. No partition function is needed, because every factor is a normalised distribution.
4. Sampling needs 3 sequential steps: draw $x_1$ from $\sigma(0.4)$, then compute $\sigma(-0.5+1.5x_1)$ and draw $x_2$, then compute the third logit from both and draw $x_3$. The most likely outcome, $(1,1,1)$ with $0.3855$, is reached only if every draw goes that way.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class MaskedLinear(nn.Linear):
    def __init__(self, n_in, n_out, mask):
        super().__init__(n_in, n_out)
        self.register_buffer("mask", mask.float())          # (n_out, n_in), 1 = connection allowed
    def forward(self, x):
        return F.linear(x, self.weight * self.mask, self.bias)

class MADE(nn.Module):
    """Binary MADE: one forward pass gives all D conditionals p(x_i = 1 | x_<i)."""
    def __init__(self, D, H=128, seed=0):
        super().__init__()
        g = torch.Generator().manual_seed(seed)
        m_in = torch.arange(1, D + 1)                        # input degrees 1..D (natural order)
        m_h = torch.randint(1, D, (H,), generator=g)         # hidden degrees in 1..D-1
        M1 = m_h[:, None] >= m_in[None, :]                   # hidden k sees x_d iff m(k) >= d
        M2 = m_h[:, None] >= m_h[None, :]
        M3 = m_in[:, None] > m_h[None, :]                    # output d sees hidden k iff d > m(k)
        self.net = nn.Sequential(MaskedLinear(D, H, M1), nn.ReLU(), MaskedLinear(H, H, M2), nn.ReLU(), MaskedLinear(H, D, M3))
        self.D = D

    def forward(self, x):                                    # x (N, D) in {0, 1}
        return self.net(x)                                   # logits (N, D)

    def nll(self, x):                                        # exact −log p(x) in nats, teacher forcing
        return F.binary_cross_entropy_with_logits(self(x), x, reduction="none").sum(-1)

    @torch.no_grad()
    def sample(self, n):                                     # D sequential passes
        x = torch.zeros(n, self.D)
        for d in range(self.D):
            x[:, d] = torch.bernoulli(torch.sigmoid(self(x)[:, d]))
        return x

torch.manual_seed(0)
D = 16
model = MADE(D)
x = (torch.rand(256, D) < 0.3).float()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for step in range(200):
    loss = model.nll(x).mean()
    opt.zero_grad(); loss.backward(); opt.step()
print(model(x).shape, model.nll(x).shape)                   # torch.Size([256, 16]) torch.Size([256])

# autoregressive check: output d must not depend on inputs d, d+1, ...
J = torch.autograd.functional.jacobian(lambda v: model(v[None])[0], torch.rand(D))
print(torch.equal(J.abs().gt(0), torch.tril(J.abs().gt(0), -1)))   # True: Jacobian strictly lower-triangular

# the conditionals multiply to a normalised distribution: sum p(x) over all 2^4 inputs of a D = 4 model
small = MADE(4)
allx = torch.tensor([[(i >> k) & 1 for k in range(4)] for i in range(16)]).float()
print(torch.exp(-small.nll(allx)).sum().item())            # 1.0 (up to float error)
s = model.sample(5)
print(s.shape)                                               # torch.Size([5, 16])
```

A binary MADE: masked layers, exact NLL with teacher forcing, a causality check and D-step sampling

> **Any order is exact; only capacity makes it matter:** The chain rule is valid in every one of the $D!$ orderings, so with fully flexible conditionals the ordering is irrelevant (KL = 0 in both orders in the figure). With a restricted conditional family it is not: the Gaussian-conditional model on two moons loses 0.272 nats in one order and 0.433 in the other. Images use raster order and text uses reading order by convention. MADE and order-agnostic training average over orderings, and MaskGIT-style models drop a fixed order entirely.

> **Continuous data needs dequantisation:** A continuous density can put arbitrarily high density on the 256 discrete values of 8-bit pixels, so its "likelihood" on raw integer data is unbounded. Either use a discrete conditional (softmax over 256 levels, or the discretised logistic mixture of PixelCNN++), or add uniform noise $u\sim U[0,1)$ to each integer and divide by 256 before evaluating a continuous density. Only then are bits/dim comparable across papers.

> **Temperature and top-k are not the model:** Ancestral sampling with the raw conditionals draws from $p_\theta$ exactly. Language models usually sample with temperature, top-$k$ or nucleus truncation, which changes every conditional and therefore the joint. These tricks trade diversity for quality. They are decoding choices, not part of the likelihood that was trained.

**Strengths**

- Exact, tractable log-likelihood; training is plain cross-entropy with no bound, no discriminator and no posterior
- Teacher forcing evaluates all conditionals in one parallel pass
- Universal: categorical conditionals can represent any discrete distribution
- Stable training that scales predictably with data and parameters (decoder-only language models)

**Limitations**

- Sampling needs $D$ sequential passes; latency grows with sequence length
- A fixed ordering is arbitrary for images and other non-sequential data
- Exposure bias: trained on true prefixes, sampled on its own
- No low-dimensional latent code for representation learning or editing

**Question:** Why can an autoregressive model be trained on a whole sequence in one forward pass but not sampled in one?

- A. Training uses a lower bound that is cheaper to compute
- B. During training the true prefix of every position is known and a causal mask enforces x<ᵢ dependence; during sampling xᵢ must be drawn before the next conditional can be computed
- C. Sampling needs a separate decoder network
- D. The softmax cannot be computed in parallel

**Answer:** B. During training the true prefix of every position is known and a causal mask enforces x<ᵢ dependence; during sampling xᵢ must be drawn before the next conditional can be computed — Teacher forcing feeds the ground truth, so all $D$ conditionals $p(x_i\mid x_{<i})$ are available at once and the mask only blocks the future. At sampling time the prefix is the model's own output, so the $D$ draws are inherently sequential.

**Question:** In a MADE with D = 4 inputs, a hidden unit has degree m(k) = 2. Which inputs can reach it and which outputs can it feed?

- A. Inputs x₁, x₂; outputs for x₃, x₄
- B. Inputs x₁, x₂; outputs for x₂, x₃, x₄
- C. Inputs x₃, x₄; outputs for x₁, x₂
- D. All inputs; outputs for x₃, x₄

**Answer:** A. Inputs x₁, x₂; outputs for x₃, x₄ — Input mask $\mathbb 1[m(k)\ge d]$ keeps $d=1,2$. Output mask $\mathbb 1[d>m(k)]$ keeps $d=3,4$. Output 3 therefore receives information from $x_1,x_2$ only through this unit, which is allowed. Output 2 must not see $x_2$, hence the strict inequality.

**Key points**

- Chain rule: $p_\theta(x)=\prod_{i=1}^{D}p_\theta(x_i\mid x_{<i})$ is exact for any ordering; training minimises the exact NLL $=H(p_{\text{data}})+\mathrm{KL}(p_{\text{data}}\|p_\theta)$.
- Teacher forcing evaluates all $D$ conditionals in one masked pass; ancestral sampling needs $D$ sequential passes.
- Causality by masking: MADE masks $\mathbb 1[m(k)\ge d]$ and $\mathbb 1[d>m(k)]$, masked convolutions (PixelCNN), dilated causal convolutions (WaveNet), causal self-attention (decoder-only LMs).
- Images are scored in bits/dim $=\mathcal L_{\text{nats}}/(D\ln2)$; uniform over 256 levels is 8, PixelCNN-family models about 3 on CIFAR-10.
- With limited conditionals the ordering matters; exposure bias and sequential sampling cost are the main weaknesses.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
