# Autoregressive image generation

> Autoregressive image models turn an image into a sequence of pixels or of discrete tokens from a learned tokenizer and predict it one element at a time. Raster order costs one network pass per token; MaskGIT and VAR cut this to about 10 passes by predicting many tokens at once.

Source: https://www.mlfoundry.in/learn/autoregressive-image-generation  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**Autoregressive image generation:** The family of image generators that apply the chain rule $p(x)=\prod_i p(x_i\mid x_{<i})$ (see **Autoregressive generative models**) to an image. The elements $x_i$ are either **pixels** (sub-pixel intensities in raster order, as in PixelRNN, PixelCNN and the Image Transformer) or **discrete tokens** of a learned tokenizer such as VQ-VAE, VQGAN or a dVAE. In the token case a transformer prior generates the token grid and the tokenizer's decoder turns it into pixels. Variants change *what* is predicted per step: one token (raster order), a confidence-selected set of tokens (MaskGIT), or a whole coarser-to-finer token map (VAR).

Van den Oord, Kalchbrenner and Kavukcuoglu's "Pixel Recurrent Neural Networks" (ICML 2016 best paper) showed that a deep network predicting one sub-pixel at a time with a 256-way softmax gives the best likelihoods on natural images. It introduced PixelRNN (Row LSTM, Diagonal BiLSTM) and the faster, fully convolutional **PixelCNN**. The same year, "Conditional Image Generation with PixelCNN Decoders" (van den Oord et al., NeurIPS 2016) fixed PixelCNN's blind spot with the **Gated PixelCNN**. Pixel-level models are slow to sample and spend capacity on imperceptible detail. From 2017 the field moved to a two-stage recipe: compress the image into a short grid of codebook indices (see **Vector-quantized VAE (VQ-VAE)**), then model that grid autoregressively. VQ-VAE-2 (2019), VQGAN "Taming Transformers" (Esser, Rombach and Ommer, CVPR 2021), DALL·E (Ramesh et al., ICML 2021) and Parti (Yu et al., 2022) follow this recipe. MaskGIT (Chang et al., CVPR 2022) and VAR (Tian et al., NeurIPS 2024) then attacked the sequential cost.

## Pixels in raster order: PixelCNN

$$
p(x)=\prod_{i=1}^{n^2}p(x_{i,R}\mid x_{<i})\;p(x_{i,G}\mid x_{<i},x_{i,R})\;p(x_{i,B}\mid x_{<i},x_{i,R},x_{i,G})
$$

- $i$: pixel index in raster order (row by row, left to right) on an $n\times n$ image
- $x_{i,R},x_{i,G},x_{i,B}$: the three colour channels, each a 256-way categorical; G is conditioned on R, and B on R and G

A PixelCNN enforces the ordering with **masked convolutions**. In a $k\times k$ kernel every weight below the centre row, and right of the centre in the centre row, is zeroed. The first layer uses **mask A**, which also zeroes the centre, so the output at $i$ never sees $x_i$. Later layers use **mask B**, which keeps the centre: a feature at $i$ may use earlier features computed at $i$, since those already exclude $x_i$. Channel masks split the feature maps into R, G and B groups to give the within-pixel ordering. Training is one convolutional pass over the image with teacher forcing. Sampling needs one pass per sub-pixel: $3n^2$ passes, 3072 for a $32\times32$ image.

Stacking masked $3\times3$ kernels leaves a **blind spot**. Each layer extends the receptive field one pixel up-left, up and up-right, and one pixel left in the current row. The up-right growth starts from the row above, so the union is a staircase: pixels above and to the right of the target, though earlier in raster order, are never seen. The **Gated PixelCNN** splits the network into a **vertical stack** that sees every row above (its kernels cover only rows above the current one) and a **horizontal stack** that sees pixels to the left in the current row, with the vertical stack feeding the horizontal one. It also replaces ReLU by a gated unit and adds conditioning:

$$
y=\tanh\big(W_{k,f}*x+V_{k,f}^\top h\big)\odot\sigma\big(W_{k,g}*x+V_{k,g}^\top h\big)
$$

- $W_{k,f},W_{k,g}$: masked convolution weights of layer $k$ for the filter and the gate
- $h$: an optional conditioning vector (class label, embedding), broadcast over all positions
- $\odot$: element-wise product; the sigmoid gate plays the role of an LSTM gate

PixelCNN++ (Salimans et al., ICLR 2017) replaced the 256-way softmax by a discretised mixture of logistics and added downsampling and shortcut connections. The Image Transformer (Parmar et al., ICML 2018) used local self-attention over pixel blocks, and the Sparse Transformer (Child et al., 2019) used factorised sparse attention to reach long contexts. ImageGPT (Chen et al., ICML 2020) trained a GPT-2 on $32\times32$ to $64\times64$ images with a 9-bit (512-colour) palette, mainly to study representation learning.

CIFAR-10 test NLL in bits/dim (lower is better), as reported in each paper

| model | paper | bits/dim |
| --- | --- | --- |
| PixelCNN | van den Oord et al., ICML 2016 | 3.14 |
| Row LSTM | van den Oord et al., ICML 2016 | 3.07 |
| Diagonal BiLSTM | van den Oord et al., ICML 2016 | 3.00 |
| Gated PixelCNN | van den Oord et al., NeurIPS 2016 | 3.03 |
| PixelCNN++ | Salimans et al., ICLR 2017 | 2.92 |
| Image Transformer (1D local) | Parmar et al., ICML 2018 | 2.90 |
| Sparse Transformer | Child et al., 2019 | 2.80 |

## Tokens instead of pixels: the two-stage recipe

A tokenizer with downsampling factor $f$ maps an $H\times W$ image to an $(H/f)\times(W/f)$ grid of indices into a codebook of size $K$. For a $256\times256$ image, $f=16$ gives 256 tokens and $f=8$ gives 1024, against 196,608 sub-pixels. The transformer prior is trained with next-token cross-entropy on the flattened grid, conditioned on a class token or on text.

**Token-based autoregressive systems**

- **VQ-VAE-2:** Razavi, van den Oord and Vinyals, NeurIPS 2019; hierarchical codes ($32\times32$ top, $64\times64$ bottom for $256\times256$ ImageNet), PixelCNN priors with self-attention
- **VQGAN:** Esser, Rombach and Ommer, CVPR 2021; VQ tokenizer trained with perceptual and patch-adversarial losses, typically $f=16$; GPT-2-style prior, sliding attention window for megapixel images
- **DALL·E:** Ramesh et al., ICML 2021; dVAE maps $256\times256$ images to $32\times32$ tokens from 8192 codes; up to 256 BPE text tokens then 1024 image tokens in one 12B-parameter decoder-only transformer, trained on 250M text–image pairs
- **Parti:** Yu et al., 2022; ViT-VQGAN tokenizer (8192 codes, 1024 tokens per $256\times256$ image), encoder–decoder transformer up to 20B parameters; zero-shot MS-COCO FID 7.23
- **MaskGIT:** Chang et al., CVPR 2022; bidirectional transformer over VQGAN tokens, 8 decoding steps; ImageNet $256\times256$ FID 6.18
- **VAR:** Tian et al., NeurIPS 2024; next-scale prediction over 10 token maps ($1^2$ to $16^2$, 680 tokens); abstract reports ImageNet $256\times256$ FID 1.73 and about 20× faster inference than their raster-order baseline

## The cost of raster order

With $N$ tokens, raster decoding takes $N$ sequential forward passes. With a KV cache, pass $k$ has one query attending to $k$ keys, so the total attention work is $\sum_{k=1}^{N}k=N(N+1)/2$ query–key pairs. That is small, but the passes are sequential and each is memory-bound on a GPU, so latency grows linearly with $N$. Raster order is also an arbitrary choice for images: the first rows are generated without any information about the rest of the scene.

## MaskGIT: parallel decoding with a mask schedule

MaskGIT trains a **bidirectional** transformer like BERT. For each image a mask ratio $\gamma(r)$ with $r\sim U[0,1)$ is drawn, that many tokens are replaced by `[MASK]`, and the loss is cross-entropy on the masked positions. To generate, it starts from an all-masked grid and runs $T$ passes. Each pass predicts every masked token in parallel, samples them, and keeps the most confident. The rest are masked again, according to the schedule:

$$
n_t=\Big\lceil\gamma\big(\tfrac{t}{T}\big)\,N\Big\rceil,\qquad \gamma(r)=\cos\big(\tfrac{\pi}{2}r\big),\qquad t=1,\dots,T
$$

- $n_t$: number of tokens still masked after pass $t$; $n_T=0$
- $\gamma$: mask schedule, decreasing from 1 to 0; the paper found the cosine schedule best among those tried
- $T$: number of decoding passes, 8 by default (against $N=256$ for raster order)
- $\text{confidence}$: the sampled token's probability, with annealed random noise so that early choices are not purely greedy

The cosine schedule reveals few tokens early, when the prediction is most uncertain, and many at the end, when most context is fixed. Each pass is full bidirectional attention over all $N$ tokens and there is no KV cache, so attention work is $T\cdot N^2$, more than raster order. But only $T$ passes are sequential.

## VAR: next-scale prediction

VAR changes the unit of autoregression from a token to a **token map**. A multi-scale VQ tokenizer represents the image as $K$ residual token maps $r_1,\dots,r_K$ of increasing side, sharing one codebook. Each map encodes what the upsampled sum of the coarser maps still misses. The transformer predicts all tokens of $r_k$ in one pass, given every coarser map:

$$
p(r_1,\dots,r_K)=\prod_{k=1}^{K}p\big(r_k\mid r_1,\dots,r_{k-1}\big),\qquad \text{sides }1,2,3,4,5,6,8,10,13,16\ \Rightarrow\ \sum_k h_k^2=680\ \text{tokens}
$$

- $r_k$: an $h_k\times h_k$ map of codebook indices; within a map the tokens are predicted in parallel
- $\text{mask}$: block-causal attention: queries of scale $k$ see all tokens of scales $1..k$

Generation takes $K=10$ passes for a $256\times256$ image. The order is coarse-to-fine, the way images are usually described, rather than top-to-bottom. The authors also report power-law scaling of test loss with model size, like that of language models.

> Figure: A 16 × 16 token grid, as for a 256 × 256 image through an f = 16 tokenizer (N = 256). Raster decoding fills it in 256 sequential passes. With a KV cache it needs Σk = 32,896 query–key pairs in total. MaskGIT with T = 8 reveals 4, 15, 24, 31, 39, 45, 48 and 50 tokens per pass (the cosine schedule leaves 252, 237, 213, 182, 143, 98, 50 and 0 masked). Each pass attends over all 256 tokens, so the total is 8 × 256² = 524,288 pairs, 15.9× raster, in 32× fewer sequential passes. At T = 4 the counts are 19, 55, 84 and 98 (262,144 pairs). At T = 16 the first pass reveals a single token. VAR produces token maps of side 1 to 16, 680 tokens in 10 passes (68 tokens per pass), with 286,434 block-causal pairs (8.7× raster). The last scale alone is 256 queries × 680 keys = 174,080 pairs. The receptive-field buttons trace which input pixels can influence pixel (9, 8). After 4 PixelCNN layers (mask A, then mask B) it sees 34 pixels and misses 6 earlier pixels up and to the right (the blind spot). After 8 layers it sees 115 and misses 21. Gated PixelCNN's vertical and horizontal stacks see 40 pixels after 4 layers and 136 after 8, with no blind spot.

## Worked example: sequence length and the MaskGIT schedule

Sequential passes and attention pairs for one 256 × 256 image (computed in Node)

| representation | sequence length N | raster passes | KV-cached pairs N(N+1)/2 |
| --- | --- | --- | --- |
| sub-pixels (PixelCNN-style) | 196,608 | 196,608 | 19,327,451,136 |
| f = 8 tokens (DALL·E, Parti) | 1024 | 1024 | 524,800 |
| f = 16 tokens (VQGAN, MaskGIT) | 256 | 256 | 32,896 |

**MaskGIT with N = 256 and T = 8**

1. Pass 1: $\gamma(1/8)=\cos(\pi/16)=0.9808$, $0.9808\cdot256=251.08$, so $n_1=252$ tokens stay masked and $256-252=4$ are kept.
2. Pass 2: $\cos(\pi/8)=0.9239$, $n_2=\lceil236.51\rceil=237$, 15 more kept. Pass 4: $\cos(\pi/4)=0.7071$, $n_4=\lceil181.02\rceil=182$, so after half the passes only 74 tokens (29%) are fixed.
3. Passes 5 to 8 leave 143, 98, 50 and 0 masked, revealing 39, 45, 48 and 50 tokens. All 256 tokens are done in 8 passes instead of 256.
4. Attention work: $8\cdot256^2=524{,}288$ pairs against $32{,}896$ for cached raster decoding. MaskGIT does 15.9× more arithmetic in 32× fewer sequential steps. Latency improves because a pass over 256 tokens costs about as much wall-clock time as a pass over one.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

class MaskedConv2d(nn.Conv2d):
    """PixelCNN mask: type A (first layer) hides the centre pixel, type B (later layers) keeps it."""
    def __init__(self, kind, c_in, c_out, k):
        super().__init__(c_in, c_out, k, padding=k // 2)
        m = torch.ones_like(self.weight)
        m[:, :, k // 2, k // 2 + (kind == "B"):] = 0         # centre row: right of centre (and centre for A)
        m[:, :, k // 2 + 1:] = 0                             # rows below
        self.register_buffer("mask", m)
    def forward(self, x):
        return F.conv2d(x, self.weight * self.mask, self.bias, padding=self.padding)

class PixelCNN(nn.Module):
    def __init__(self, layers=4, ch=32, levels=256):
        super().__init__()
        mods = [MaskedConv2d("A", 1, ch, 3)]
        for _ in range(layers - 1):
            mods += [nn.ReLU(), MaskedConv2d("B", ch, ch, 3)]
        mods += [nn.ReLU(), nn.Conv2d(ch, levels, 1)]       # 256-way softmax per pixel
        self.net = nn.Sequential(*mods)
    def forward(self, x):                                    # x (N, 1, H, W) in [0, 1]
        return self.net(x)                                   # logits (N, 256, H, W)

torch.manual_seed(0)
net = PixelCNN(layers=4)
x = torch.randint(0, 256, (8, 1, 16, 16))
logits = net(x.float() / 255)
nll = F.cross_entropy(logits, x[:, 0])                       # teacher forcing: all pixels in one pass
print(logits.shape, round(nll.item() / math.log(2), 2))      # torch.Size([8, 256, 16, 16]) 8.0 bits/dim at init

# which inputs does the output at (9, 8) depend on? gradient of one logit w.r.t. the input image
inp = torch.rand(1, 1, 16, 16, requires_grad=True)
net(inp)[0, :, 9, 8].sum().backward()
seen = inp.grad[0, 0].ne(0)
print(seen.sum().item(), seen[9, 8:].any().item(), seen[8, 12].item())   # 34 False False: 34 pixels, blind spot at (8, 12)

def maskgit_schedule(N=256, T=8):
    """Tokens still masked after each of T parallel decoding passes (cosine schedule)."""
    return [math.ceil(N * math.cos(math.pi / 2 * t / T) - 1e-9) if t < T else 0 for t in range(T + 1)]
m = maskgit_schedule()
print(m, [a - b for a, b in zip(m, m[1:])])   # [256, 252, 237, 213, 182, 143, 98, 50, 0] [4, 15, 24, 31, 39, 45, 48, 50]
```

PixelCNN masked convolutions, a gradient check of the receptive field, and the MaskGIT schedule

> **Likelihood and sample quality disagree:** Pixel-level models win on bits/dim (2.80 for the Sparse Transformer on CIFAR-10), but most of those bits encode imperceptible high-frequency detail, and their samples look less coherent than GAN samples. Token-level priors give up exact pixel likelihood: the tokenizer is lossy, and quality is measured by FID or human preference. Do not compare a token model's per-token cross-entropy with a pixel model's bits/dim.

> **Pairs are not latency:** Counting query–key pairs, as the figure does, measures arithmetic. Decoding latency on accelerators is set by the number of **sequential** passes and by memory traffic per pass. Small raster passes are memory-bound and leave most of the hardware idle. A MaskGIT or VAR pass over hundreds of tokens uses it well. This is why 15.9× more arithmetic can still be faster.

> **The tokenizer sets the ceiling:** In a two-stage model the prior can never produce detail that the tokenizer's decoder cannot reconstruct. VQGAN's perceptual and adversarial losses, ViT-VQGAN's factorised $\ell_2$-normalised codes and VAR's multi-scale residual quantiser are all tokenizer improvements, and much of the progress from DALL·E to VAR came from them. Codebook collapse and code usage are covered in **Vector-quantized VAE (VQ-VAE)**.

**Strengths**

- Exact likelihood (pixel models) or a plain cross-entropy objective (token priors), stable to train and to scale
- Text conditioning is just a prefix (DALL·E) or an encoder (Parti) in a standard transformer
- MaskGIT and VAR reduce sampling to about 8–10 sequential passes
- Reuses language-model infrastructure: KV caches, tokenizers, scaling recipes

**Limitations**

- Raster order needs $N$ sequential passes and imposes an arbitrary order on a 2-D signal
- Pixel models spend capacity on imperceptible detail; token models are capped by a lossy tokenizer
- Masked convolutions have a blind spot unless built with vertical and horizontal stacks
- MaskGIT samples many tokens independently within a pass, which can produce inconsistent pairs if T is too small

**Question:** Why does a stack of masked 3 × 3 convolutions in the original PixelCNN have a blind spot?

- A. Mask A hides the centre pixel in every layer
- B. Each layer reaches one pixel further up-right only from rows already above, so the union of kernels forms a staircase that never covers some earlier pixels above and to the right
- C. The 256-way softmax cannot represent edges
- D. Padding removes the rightmost columns

**Answer:** B. Each layer reaches one pixel further up-right only from rows already above, so the union of kernels forms a staircase that never covers some earlier pixels above and to the right — The current row contributes only pixels to the left, and the up-right offset is applied from the row above. Pixels at $(r-1,c+2)$ and beyond, although earlier in raster order, are never reached: 6 such pixels after 4 layers in the figure. Gated PixelCNN's vertical stack covers all rows above and removes the blind spot.

**Question:** MaskGIT generates N = 256 tokens with T = 8 and the cosine schedule. How many tokens are still masked after pass 4?

- A. 128
- B. 182
- C. 64
- D. 213

**Answer:** B. 182 — $n_4=\lceil\cos(\pi/2\cdot4/8)\cdot256\rceil=\lceil0.7071\cdot256\rceil=\lceil181.02\rceil=182$. A linear schedule would give 128. The cosine keeps more tokens masked early, when predictions are least certain.

**Key points**

- PixelCNN: masked convolutions (mask A in layer 1 hides the centre, mask B after); $3n^2$ sequential passes per sample; Gated PixelCNN's vertical + horizontal stacks remove the blind spot.
- Two-stage recipe: tokenizer (VQ-VAE, VQGAN, dVAE) maps $256^2$ pixels to $16\times16$ or $32\times32$ codes, and a transformer prior models the codes autoregressively.
- Raster decoding: $N$ sequential passes, $N(N+1)/2$ cached query–key pairs (32,896 for $N=256$).
- MaskGIT: bidirectional transformer, $n_t=\lceil\cos(\tfrac{\pi}{2}\tfrac tT)N\rceil$ tokens remain masked after pass $t$, $T=8$ passes.
- VAR: next-scale prediction over 10 residual token maps ($1^2$ to $16^2$, 680 tokens), one pass per scale.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
