Definition
Autoregressive image generation The family of image generators that apply the chain rule (see Autoregressive generative models) to an image. The elements are either pixels (sub-pixel intensities in raster order, as in PixelRNN, PixelCNN and the Image Transformer) or discrete tokens of a learned tokenizer such as VQ-VAE, VQGAN or a dVAE. In the token case a transformer prior generates the token grid and the tokenizer's decoder turns it into pixels. Variants change what is predicted per step: one token (raster order), a confidence-selected set of tokens (MaskGIT), or a whole coarser-to-finer token map (VAR).
Van den Oord, Kalchbrenner and Kavukcuoglu's "Pixel Recurrent Neural Networks" (ICML 2016 best paper) showed that a deep network predicting one sub-pixel at a time with a 256-way softmax gives the best likelihoods on natural images. It introduced PixelRNN (Row LSTM, Diagonal BiLSTM) and the faster, fully convolutional PixelCNN. The same year, "Conditional Image Generation with PixelCNN Decoders" (van den Oord et al., NeurIPS 2016) fixed PixelCNN's blind spot with the Gated PixelCNN. Pixel-level models are slow to sample and spend capacity on imperceptible detail. From 2017 the field moved to a two-stage recipe: compress the image into a short grid of codebook indices (see Vector-quantized VAE (VQ-VAE)), then model that grid autoregressively. VQ-VAE-2 (2019), VQGAN "Taming Transformers" (Esser, Rombach and Ommer, CVPR 2021), DALL·E (Ramesh et al., ICML 2021) and Parti (Yu et al., 2022) follow this recipe. MaskGIT (Chang et al., CVPR 2022) and VAR (Tian et al., NeurIPS 2024) then attacked the sequential cost.
Pixels in raster order: PixelCNN
- pixel index in raster order (row by row, left to right) on an image
- the three colour channels, each a 256-way categorical; G is conditioned on R, and B on R and G
A PixelCNN enforces the ordering with masked convolutions. In a kernel every weight below the centre row, and right of the centre in the centre row, is zeroed. The first layer uses mask A, which also zeroes the centre, so the output at never sees . Later layers use mask B, which keeps the centre: a feature at may use earlier features computed at , since those already exclude . Channel masks split the feature maps into R, G and B groups to give the within-pixel ordering. Training is one convolutional pass over the image with teacher forcing. Sampling needs one pass per sub-pixel: passes, 3072 for a image.
Stacking masked kernels leaves a blind spot. Each layer extends the receptive field one pixel up-left, up and up-right, and one pixel left in the current row. The up-right growth starts from the row above, so the union is a staircase: pixels above and to the right of the target, though earlier in raster order, are never seen. The Gated PixelCNN splits the network into a vertical stack that sees every row above (its kernels cover only rows above the current one) and a horizontal stack that sees pixels to the left in the current row, with the vertical stack feeding the horizontal one. It also replaces ReLU by a gated unit and adds conditioning:
- masked convolution weights of layer for the filter and the gate
- an optional conditioning vector (class label, embedding), broadcast over all positions
- element-wise product; the sigmoid gate plays the role of an LSTM gate
PixelCNN++ (Salimans et al., ICLR 2017) replaced the 256-way softmax by a discretised mixture of logistics and added downsampling and shortcut connections. The Image Transformer (Parmar et al., ICML 2018) used local self-attention over pixel blocks, and the Sparse Transformer (Child et al., 2019) used factorised sparse attention to reach long contexts. ImageGPT (Chen et al., ICML 2020) trained a GPT-2 on to images with a 9-bit (512-colour) palette, mainly to study representation learning.
| model | paper | bits/dim |
|---|---|---|
| PixelCNN | van den Oord et al., ICML 2016 | 3.14 |
| Row LSTM | van den Oord et al., ICML 2016 | 3.07 |
| Diagonal BiLSTM | van den Oord et al., ICML 2016 | 3.00 |
| Gated PixelCNN | van den Oord et al., NeurIPS 2016 | 3.03 |
| PixelCNN++ | Salimans et al., ICLR 2017 | 2.92 |
| Image Transformer (1D local) | Parmar et al., ICML 2018 | 2.90 |
| Sparse Transformer | Child et al., 2019 | 2.80 |
Tokens instead of pixels: the two-stage recipe
A tokenizer with downsampling factor maps an image to an grid of indices into a codebook of size . For a image, gives 256 tokens and gives 1024, against 196,608 sub-pixels. The transformer prior is trained with next-token cross-entropy on the flattened grid, conditioned on a class token or on text.
Token-based autoregressive systems
- VQ-VAE-2
- Razavi, van den Oord and Vinyals, NeurIPS 2019; hierarchical codes ( top, bottom for ImageNet), PixelCNN priors with self-attention
- VQGAN
- Esser, Rombach and Ommer, CVPR 2021; VQ tokenizer trained with perceptual and patch-adversarial losses, typically ; GPT-2-style prior, sliding attention window for megapixel images
- DALL·E
- Ramesh et al., ICML 2021; dVAE maps images to tokens from 8192 codes; up to 256 BPE text tokens then 1024 image tokens in one 12B-parameter decoder-only transformer, trained on 250M text–image pairs
- Parti
- Yu et al., 2022; ViT-VQGAN tokenizer (8192 codes, 1024 tokens per image), encoder–decoder transformer up to 20B parameters; zero-shot MS-COCO FID 7.23
- MaskGIT
- Chang et al., CVPR 2022; bidirectional transformer over VQGAN tokens, 8 decoding steps; ImageNet FID 6.18
- VAR
- Tian et al., NeurIPS 2024; next-scale prediction over 10 token maps ( to , 680 tokens); abstract reports ImageNet FID 1.73 and about 20× faster inference than their raster-order baseline
The cost of raster order
With tokens, raster decoding takes sequential forward passes. With a KV cache, pass has one query attending to keys, so the total attention work is query–key pairs. That is small, but the passes are sequential and each is memory-bound on a GPU, so latency grows linearly with . Raster order is also an arbitrary choice for images: the first rows are generated without any information about the rest of the scene.
MaskGIT: parallel decoding with a mask schedule
MaskGIT trains a bidirectional transformer like BERT. For each image a mask ratio with is drawn, that many tokens are replaced by [MASK], and the loss is cross-entropy on the masked positions. To generate, it starts from an all-masked grid and runs passes. Each pass predicts every masked token in parallel, samples them, and keeps the most confident. The rest are masked again, according to the schedule:
- number of tokens still masked after pass ;
- mask schedule, decreasing from 1 to 0; the paper found the cosine schedule best among those tried
- number of decoding passes, 8 by default (against for raster order)
- the sampled token's probability, with annealed random noise so that early choices are not purely greedy
The cosine schedule reveals few tokens early, when the prediction is most uncertain, and many at the end, when most context is fixed. Each pass is full bidirectional attention over all tokens and there is no KV cache, so attention work is , more than raster order. But only passes are sequential.
VAR: next-scale prediction
VAR changes the unit of autoregression from a token to a token map. A multi-scale VQ tokenizer represents the image as residual token maps of increasing side, sharing one codebook. Each map encodes what the upsampled sum of the coarser maps still misses. The transformer predicts all tokens of in one pass, given every coarser map:
- an map of codebook indices; within a map the tokens are predicted in parallel
- block-causal attention: queries of scale see all tokens of scales
Generation takes passes for a image. The order is coarse-to-fine, the way images are usually described, rather than top-to-bottom. The authors also report power-law scaling of test loss with model size, like that of language models.
Worked example: sequence length and the MaskGIT schedule
| representation | sequence length N | raster passes | KV-cached pairs N(N+1)/2 |
|---|---|---|---|
| sub-pixels (PixelCNN-style) | 196,608 | 196,608 | 19,327,451,136 |
| f = 8 tokens (DALL·E, Parti) | 1024 | 1024 | 524,800 |
| f = 16 tokens (VQGAN, MaskGIT) | 256 | 256 | 32,896 |
MaskGIT with N = 256 and T = 8
- 1Pass 1: , , so tokens stay masked and are kept.
- 2Pass 2: , , 15 more kept. Pass 4: , , so after half the passes only 74 tokens (29%) are fixed.
- 3Passes 5 to 8 leave 143, 98, 50 and 0 masked, revealing 39, 45, 48 and 50 tokens. All 256 tokens are done in 8 passes instead of 256.
- 4Attention work: pairs against for cached raster decoding. MaskGIT does 15.9× more arithmetic in 32× fewer sequential steps. Latency improves because a pass over 256 tokens costs about as much wall-clock time as a pass over one.
1import math, torch, torch.nn as nn, torch.nn.functional as F23class MaskedConv2d(nn.Conv2d):4 """PixelCNN mask: type A (first layer) hides the centre pixel, type B (later layers) keeps it."""5 def __init__(self, kind, c_in, c_out, k):6 super().__init__(c_in, c_out, k, padding=k // 2)7 m = torch.ones_like(self.weight)8 m[:, :, k // 2, k // 2 + (kind == "B"):] = 0 # centre row: right of centre (and centre for A)9 m[:, :, k // 2 + 1:] = 0 # rows below10 self.register_buffer("mask", m)11 def forward(self, x):12 return F.conv2d(x, self.weight * self.mask, self.bias, padding=self.padding)1314class PixelCNN(nn.Module):15 def __init__(self, layers=4, ch=32, levels=256):16 super().__init__()17 mods = [MaskedConv2d("A", 1, ch, 3)]18 for _ in range(layers - 1):19 mods += [nn.ReLU(), MaskedConv2d("B", ch, ch, 3)]20 mods += [nn.ReLU(), nn.Conv2d(ch, levels, 1)] # 256-way softmax per pixel21 self.net = nn.Sequential(*mods)22 def forward(self, x): # x (N, 1, H, W) in [0, 1]23 return self.net(x) # logits (N, 256, H, W)2425torch.manual_seed(0)26net = PixelCNN(layers=4)27x = torch.randint(0, 256, (8, 1, 16, 16))28logits = net(x.float() / 255)29nll = F.cross_entropy(logits, x[:, 0]) # teacher forcing: all pixels in one pass30print(logits.shape, round(nll.item() / math.log(2), 2)) # torch.Size([8, 256, 16, 16]) 8.0 bits/dim at init3132# which inputs does the output at (9, 8) depend on? gradient of one logit w.r.t. the input image33inp = torch.rand(1, 1, 16, 16, requires_grad=True)34net(inp)[0, :, 9, 8].sum().backward()35seen = inp.grad[0, 0].ne(0)36print(seen.sum().item(), seen[9, 8:].any().item(), seen[8, 12].item()) # 34 False False: 34 pixels, blind spot at (8, 12)3738def maskgit_schedule(N=256, T=8):39 """Tokens still masked after each of T parallel decoding passes (cosine schedule)."""40 return [math.ceil(N * math.cos(math.pi / 2 * t / T) - 1e-9) if t < T else 0 for t in range(T + 1)]41m = maskgit_schedule()42print(m, [a - b for a, b in zip(m, m[1:])]) # [256, 252, 237, 213, 182, 143, 98, 50, 0] [4, 15, 24, 31, 39, 45, 48, 50]Use when
- +Exact likelihood (pixel models) or a plain cross-entropy objective (token priors), stable to train and to scale
- +Text conditioning is just a prefix (DALL·E) or an encoder (Parti) in a standard transformer
- +MaskGIT and VAR reduce sampling to about 8–10 sequential passes
- +Reuses language-model infrastructure: KV caches, tokenizers, scaling recipes
Avoid when
- −Raster order needs sequential passes and imposes an arbitrary order on a 2-D signal
- −Pixel models spend capacity on imperceptible detail; token models are capped by a lossy tokenizer
- −Masked convolutions have a blind spot unless built with vertical and horizontal stacks
- −MaskGIT samples many tokens independently within a pass, which can produce inconsistent pairs if T is too small
Check yourself
Why does a stack of masked 3 × 3 convolutions in the original PixelCNN have a blind spot?
Check yourself
MaskGIT generates N = 256 tokens with T = 8 and the cosine schedule. How many tokens are still masked after pass 4?
Remember
- ✓PixelCNN: masked convolutions (mask A in layer 1 hides the centre, mask B after); sequential passes per sample; Gated PixelCNN's vertical + horizontal stacks remove the blind spot.
- ✓Two-stage recipe: tokenizer (VQ-VAE, VQGAN, dVAE) maps pixels to or codes, and a transformer prior models the codes autoregressively.
- ✓Raster decoding: sequential passes, cached query–key pairs (32,896 for ).
- ✓MaskGIT: bidirectional transformer, tokens remain masked after pass , passes.
- ✓VAR: next-scale prediction over 10 residual token maps ( to , 680 tokens), one pass per scale.