Autoregressive image generation

Autoregressive image models turn an image into a sequence of pixels or of discrete tokens from a learned tokenizer and predict it one element at a time. Raster order costs one network pass per token; MaskGIT and VAR cut this to about 10 passes by predicting many tokens at once.

15 min read1294 wordsadvanced0 viewsupdated 2026-10-12

Definition

Autoregressive image generation The family of image generators that apply the chain rule p(x)=∏ip(xi∣x<i)p(x)=\prod_i p(x_i\mid x_{<i}) (see Autoregressive generative models) to an image. The elements xix_i are either pixels (sub-pixel intensities in raster order, as in PixelRNN, PixelCNN and the Image Transformer) or discrete tokens of a learned tokenizer such as VQ-VAE, VQGAN or a dVAE. In the token case a transformer prior generates the token grid and the tokenizer's decoder turns it into pixels. Variants change what is predicted per step: one token (raster order), a confidence-selected set of tokens (MaskGIT), or a whole coarser-to-finer token map (VAR).

Van den Oord, Kalchbrenner and Kavukcuoglu's "Pixel Recurrent Neural Networks" (ICML 2016 best paper) showed that a deep network predicting one sub-pixel at a time with a 256-way softmax gives the best likelihoods on natural images. It introduced PixelRNN (Row LSTM, Diagonal BiLSTM) and the faster, fully convolutional PixelCNN. The same year, "Conditional Image Generation with PixelCNN Decoders" (van den Oord et al., NeurIPS 2016) fixed PixelCNN's blind spot with the Gated PixelCNN. Pixel-level models are slow to sample and spend capacity on imperceptible detail. From 2017 the field moved to a two-stage recipe: compress the image into a short grid of codebook indices (see Vector-quantized VAE (VQ-VAE)), then model that grid autoregressively. VQ-VAE-2 (2019), VQGAN "Taming Transformers" (Esser, Rombach and Ommer, CVPR 2021), DALL·E (Ramesh et al., ICML 2021) and Parti (Yu et al., 2022) follow this recipe. MaskGIT (Chang et al., CVPR 2022) and VAR (Tian et al., NeurIPS 2024) then attacked the sequential cost.

Pixels in raster order: PixelCNN

p(x)=∏i=1n2p(xi,R∣x<i)  p(xi,G∣x<i,xi,R)  p(xi,B∣x<i,xi,R,xi,G)p(x)=\prod_{i=1}^{n^2}p(x_{i,R}\mid x_{<i})\;p(x_{i,G}\mid x_{<i},x_{i,R})\;p(x_{i,B}\mid x_{<i},x_{i,R},x_{i,G})
ii
pixel index in raster order (row by row, left to right) on an n×nn\times n image
xi,R,xi,G,xi,Bx_{i,R},x_{i,G},x_{i,B}
the three colour channels, each a 256-way categorical; G is conditioned on R, and B on R and G

A PixelCNN enforces the ordering with masked convolutions. In a k×kk\times k kernel every weight below the centre row, and right of the centre in the centre row, is zeroed. The first layer uses mask A, which also zeroes the centre, so the output at ii never sees xix_i. Later layers use mask B, which keeps the centre: a feature at ii may use earlier features computed at ii, since those already exclude xix_i. Channel masks split the feature maps into R, G and B groups to give the within-pixel ordering. Training is one convolutional pass over the image with teacher forcing. Sampling needs one pass per sub-pixel: 3n23n^2 passes, 3072 for a 32×3232\times32 image.

Stacking masked 3×33\times3 kernels leaves a blind spot. Each layer extends the receptive field one pixel up-left, up and up-right, and one pixel left in the current row. The up-right growth starts from the row above, so the union is a staircase: pixels above and to the right of the target, though earlier in raster order, are never seen. The Gated PixelCNN splits the network into a vertical stack that sees every row above (its kernels cover only rows above the current one) and a horizontal stack that sees pixels to the left in the current row, with the vertical stack feeding the horizontal one. It also replaces ReLU by a gated unit and adds conditioning:

y=tanh⁡(Wk,f∗x+Vk,f⊤h)⊙σ(Wk,g∗x+Vk,g⊤h)y=\tanh\big(W_{k,f}*x+V_{k,f}^\top h\big)\odot\sigma\big(W_{k,g}*x+V_{k,g}^\top h\big)
Wk,f,Wk,gW_{k,f},W_{k,g}
masked convolution weights of layer kk for the filter and the gate
hh
an optional conditioning vector (class label, embedding), broadcast over all positions
⊙\odot
element-wise product; the sigmoid gate plays the role of an LSTM gate

PixelCNN++ (Salimans et al., ICLR 2017) replaced the 256-way softmax by a discretised mixture of logistics and added downsampling and shortcut connections. The Image Transformer (Parmar et al., ICML 2018) used local self-attention over pixel blocks, and the Sparse Transformer (Child et al., 2019) used factorised sparse attention to reach long contexts. ImageGPT (Chen et al., ICML 2020) trained a GPT-2 on 32×3232\times32 to 64×6464\times64 images with a 9-bit (512-colour) palette, mainly to study representation learning.

modelpaperbits/dim
PixelCNNvan den Oord et al., ICML 20163.14
Row LSTMvan den Oord et al., ICML 20163.07
Diagonal BiLSTMvan den Oord et al., ICML 20163.00
Gated PixelCNNvan den Oord et al., NeurIPS 20163.03
PixelCNN++Salimans et al., ICLR 20172.92
Image Transformer (1D local)Parmar et al., ICML 20182.90
Sparse TransformerChild et al., 20192.80
CIFAR-10 test NLL in bits/dim (lower is better), as reported in each paper

Tokens instead of pixels: the two-stage recipe

A tokenizer with downsampling factor ff maps an H×WH\times W image to an (H/f)×(W/f)(H/f)\times(W/f) grid of indices into a codebook of size KK. For a 256×256256\times256 image, f=16f=16 gives 256 tokens and f=8f=8 gives 1024, against 196,608 sub-pixels. The transformer prior is trained with next-token cross-entropy on the flattened grid, conditioned on a class token or on text.

Token-based autoregressive systems

VQ-VAE-2
Razavi, van den Oord and Vinyals, NeurIPS 2019; hierarchical codes (32×3232\times32 top, 64×6464\times64 bottom for 256×256256\times256 ImageNet), PixelCNN priors with self-attention
VQGAN
Esser, Rombach and Ommer, CVPR 2021; VQ tokenizer trained with perceptual and patch-adversarial losses, typically f=16f=16; GPT-2-style prior, sliding attention window for megapixel images
DALL·E
Ramesh et al., ICML 2021; dVAE maps 256×256256\times256 images to 32×3232\times32 tokens from 8192 codes; up to 256 BPE text tokens then 1024 image tokens in one 12B-parameter decoder-only transformer, trained on 250M text–image pairs
Parti
Yu et al., 2022; ViT-VQGAN tokenizer (8192 codes, 1024 tokens per 256×256256\times256 image), encoder–decoder transformer up to 20B parameters; zero-shot MS-COCO FID 7.23
MaskGIT
Chang et al., CVPR 2022; bidirectional transformer over VQGAN tokens, 8 decoding steps; ImageNet 256×256256\times256 FID 6.18
VAR
Tian et al., NeurIPS 2024; next-scale prediction over 10 token maps (121^2 to 16216^2, 680 tokens); abstract reports ImageNet 256×256256\times256 FID 1.73 and about 20× faster inference than their raster-order baseline

The cost of raster order

With NN tokens, raster decoding takes NN sequential forward passes. With a KV cache, pass kk has one query attending to kk keys, so the total attention work is ∑k=1Nk=N(N+1)/2\sum_{k=1}^{N}k=N(N+1)/2 query–key pairs. That is small, but the passes are sequential and each is memory-bound on a GPU, so latency grows linearly with NN. Raster order is also an arbitrary choice for images: the first rows are generated without any information about the rest of the scene.

MaskGIT: parallel decoding with a mask schedule

MaskGIT trains a bidirectional transformer like BERT. For each image a mask ratio γ(r)\gamma(r) with r∼U[0,1)r\sim U[0,1) is drawn, that many tokens are replaced by [MASK], and the loss is cross-entropy on the masked positions. To generate, it starts from an all-masked grid and runs TT passes. Each pass predicts every masked token in parallel, samples them, and keeps the most confident. The rest are masked again, according to the schedule:

nt=⌈γ(tT) N⌉,γ(r)=cos⁡(π2r),t=1,…,Tn_t=\Big\lceil\gamma\big(\tfrac{t}{T}\big)\,N\Big\rceil,\qquad \gamma(r)=\cos\big(\tfrac{\pi}{2}r\big),\qquad t=1,\dots,T
ntn_t
number of tokens still masked after pass tt; nT=0n_T=0
γ\gamma
mask schedule, decreasing from 1 to 0; the paper found the cosine schedule best among those tried
TT
number of decoding passes, 8 by default (against N=256N=256 for raster order)
confidence\text{confidence}
the sampled token's probability, with annealed random noise so that early choices are not purely greedy

The cosine schedule reveals few tokens early, when the prediction is most uncertain, and many at the end, when most context is fixed. Each pass is full bidirectional attention over all NN tokens and there is no KV cache, so attention work is T⋅N2T\cdot N^2, more than raster order. But only TT passes are sequential.

VAR: next-scale prediction

VAR changes the unit of autoregression from a token to a token map. A multi-scale VQ tokenizer represents the image as KK residual token maps r1,…,rKr_1,\dots,r_K of increasing side, sharing one codebook. Each map encodes what the upsampled sum of the coarser maps still misses. The transformer predicts all tokens of rkr_k in one pass, given every coarser map:

p(r1,…,rK)=∏k=1Kp(rk∣r1,…,rk−1),sides 1,2,3,4,5,6,8,10,13,16 ⇒ ∑khk2=680 tokensp(r_1,\dots,r_K)=\prod_{k=1}^{K}p\big(r_k\mid r_1,\dots,r_{k-1}\big),\qquad \text{sides }1,2,3,4,5,6,8,10,13,16\ \Rightarrow\ \sum_k h_k^2=680\ \text{tokens}
rkr_k
an hk×hkh_k\times h_k map of codebook indices; within a map the tokens are predicted in parallel
mask\text{mask}
block-causal attention: queries of scale kk see all tokens of scales 1..k1..k

Generation takes K=10K=10 passes for a 256×256256\times256 image. The order is coarse-to-fine, the way images are usually described, rather than top-to-bottom. The authors also report power-law scaling of test loss with model size, like that of language models.

MaskGIT: predict all masked tokens in parallel, keep the most confident16×16 tokens · grey = masked / not yet generatedforward pass3 of 8tokens this pass24tokens fixed43 of 256attention pairs so far196.6ktotal pairs524.3k (15.9× raster)sequential passes8 (32.0 tok/pass)415243139454850tokens revealed per pass: few early (uncertain), many latestill masked after pass t: ⌈N·cos(πt/2T)⌉ = 213 of 256each pass is full bidirectional attention over all 256 tokens: N² pairs□ produced by the current pass · shade = token value
decoding
receptive field
fig.A 16 × 16 token grid, as for a 256 × 256 image through an f = 16 tokenizer (N = 256). Raster decoding fills it in 256 sequential passes. With a KV cache it needs Σk = 32,896 query–key pairs in total. MaskGIT with T = 8 reveals 4, 15, 24, 31, 39, 45, 48 and 50 tokens per pass (the cosine schedule leaves 252, 237, 213, 182, 143, 98, 50 and 0 masked). Each pass attends over all 256 tokens, so the total is 8 × 256² = 524,288 pairs, 15.9× raster, in 32× fewer sequential passes. At T = 4 the counts are 19, 55, 84 and 98 (262,144 pairs). At T = 16 the first pass reveals a single token. VAR produces token maps of side 1 to 16, 680 tokens in 10 passes (68 tokens per pass), with 286,434 block-causal pairs (8.7× raster). The last scale alone is 256 queries × 680 keys = 174,080 pairs. The receptive-field buttons trace which input pixels can influence pixel (9, 8). After 4 PixelCNN layers (mask A, then mask B) it sees 34 pixels and misses 6 earlier pixels up and to the right (the blind spot). After 8 layers it sees 115 and misses 21. Gated PixelCNN's vertical and horizontal stacks see 40 pixels after 4 layers and 136 after 8, with no blind spot.

Worked example: sequence length and the MaskGIT schedule

representationsequence length Nraster passesKV-cached pairs N(N+1)/2
sub-pixels (PixelCNN-style)196,608196,60819,327,451,136
f = 8 tokens (DALL·E, Parti)10241024524,800
f = 16 tokens (VQGAN, MaskGIT)25625632,896
Sequential passes and attention pairs for one 256 × 256 image (computed in Node)

MaskGIT with N = 256 and T = 8

  1. 1Pass 1: γ(1/8)=cos⁡(π/16)=0.9808\gamma(1/8)=\cos(\pi/16)=0.9808, 0.9808⋅256=251.080.9808\cdot256=251.08, so n1=252n_1=252 tokens stay masked and 256−252=4256-252=4 are kept.
  2. 2Pass 2: cos⁡(π/8)=0.9239\cos(\pi/8)=0.9239, n2=⌈236.51⌉=237n_2=\lceil236.51\rceil=237, 15 more kept. Pass 4: cos⁡(π/4)=0.7071\cos(\pi/4)=0.7071, n4=⌈181.02⌉=182n_4=\lceil181.02\rceil=182, so after half the passes only 74 tokens (29%) are fixed.
  3. 3Passes 5 to 8 leave 143, 98, 50 and 0 masked, revealing 39, 45, 48 and 50 tokens. All 256 tokens are done in 8 passes instead of 256.
  4. 4Attention work: 8⋅2562=524,2888\cdot256^2=524{,}288 pairs against 32,89632{,}896 for cached raster decoding. MaskGIT does 15.9× more arithmetic in 32× fewer sequential steps. Latency improves because a pass over 256 tokens costs about as much wall-clock time as a pass over one.
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3class MaskedConv2d(nn.Conv2d):
4 """PixelCNN mask: type A (first layer) hides the centre pixel, type B (later layers) keeps it."""
5 def __init__(self, kind, c_in, c_out, k):
6 super().__init__(c_in, c_out, k, padding=k // 2)
7 m = torch.ones_like(self.weight)
8 m[:, :, k // 2, k // 2 + (kind == "B"):] = 0 # centre row: right of centre (and centre for A)
9 m[:, :, k // 2 + 1:] = 0 # rows below
10 self.register_buffer("mask", m)
11 def forward(self, x):
12 return F.conv2d(x, self.weight * self.mask, self.bias, padding=self.padding)
13
14class PixelCNN(nn.Module):
15 def __init__(self, layers=4, ch=32, levels=256):
16 super().__init__()
17 mods = [MaskedConv2d("A", 1, ch, 3)]
18 for _ in range(layers - 1):
19 mods += [nn.ReLU(), MaskedConv2d("B", ch, ch, 3)]
20 mods += [nn.ReLU(), nn.Conv2d(ch, levels, 1)] # 256-way softmax per pixel
21 self.net = nn.Sequential(*mods)
22 def forward(self, x): # x (N, 1, H, W) in [0, 1]
23 return self.net(x) # logits (N, 256, H, W)
24
25torch.manual_seed(0)
26net = PixelCNN(layers=4)
27x = torch.randint(0, 256, (8, 1, 16, 16))
28logits = net(x.float() / 255)
29nll = F.cross_entropy(logits, x[:, 0]) # teacher forcing: all pixels in one pass
30print(logits.shape, round(nll.item() / math.log(2), 2)) # torch.Size([8, 256, 16, 16]) 8.0 bits/dim at init
31
32# which inputs does the output at (9, 8) depend on? gradient of one logit w.r.t. the input image
33inp = torch.rand(1, 1, 16, 16, requires_grad=True)
34net(inp)[0, :, 9, 8].sum().backward()
35seen = inp.grad[0, 0].ne(0)
36print(seen.sum().item(), seen[9, 8:].any().item(), seen[8, 12].item()) # 34 False False: 34 pixels, blind spot at (8, 12)
37
38def maskgit_schedule(N=256, T=8):
39 """Tokens still masked after each of T parallel decoding passes (cosine schedule)."""
40 return [math.ceil(N * math.cos(math.pi / 2 * t / T) - 1e-9) if t < T else 0 for t in range(T + 1)]
41m = maskgit_schedule()
42print(m, [a - b for a, b in zip(m, m[1:])]) # [256, 252, 237, 213, 182, 143, 98, 50, 0] [4, 15, 24, 31, 39, 45, 48, 50]
PixelCNN masked convolutions, a gradient check of the receptive field, and the MaskGIT schedule

Use when

  • +Exact likelihood (pixel models) or a plain cross-entropy objective (token priors), stable to train and to scale
  • +Text conditioning is just a prefix (DALL·E) or an encoder (Parti) in a standard transformer
  • +MaskGIT and VAR reduce sampling to about 8–10 sequential passes
  • +Reuses language-model infrastructure: KV caches, tokenizers, scaling recipes

Avoid when

  • −Raster order needs NN sequential passes and imposes an arbitrary order on a 2-D signal
  • −Pixel models spend capacity on imperceptible detail; token models are capped by a lossy tokenizer
  • −Masked convolutions have a blind spot unless built with vertical and horizontal stacks
  • −MaskGIT samples many tokens independently within a pass, which can produce inconsistent pairs if T is too small

Check yourself

Why does a stack of masked 3 × 3 convolutions in the original PixelCNN have a blind spot?

Check yourself

MaskGIT generates N = 256 tokens with T = 8 and the cosine schedule. How many tokens are still masked after pass 4?

Remember

  • ✓PixelCNN: masked convolutions (mask A in layer 1 hides the centre, mask B after); 3n23n^2 sequential passes per sample; Gated PixelCNN's vertical + horizontal stacks remove the blind spot.
  • ✓Two-stage recipe: tokenizer (VQ-VAE, VQGAN, dVAE) maps 2562256^2 pixels to 16×1616\times16 or 32×3232\times32 codes, and a transformer prior models the codes autoregressively.
  • ✓Raster decoding: NN sequential passes, N(N+1)/2N(N+1)/2 cached query–key pairs (32,896 for N=256N=256).
  • ✓MaskGIT: bidirectional transformer, nt=⌈cos⁡(π2tT)N⌉n_t=\lceil\cos(\tfrac{\pi}{2}\tfrac tT)N\rceil tokens remain masked after pass tt, T=8T=8 passes.
  • ✓VAR: next-scale prediction over 10 residual token maps (121^2 to 16216^2, 680 tokens), one pass per scale.
← back to Deep Learning roadmap