Vision Transformers (ViT)

Cut the image into 16×16 patches, embed each as a token, add a [CLS] token and position embeddings, and run a plain Transformer encoder. Weak image priors mean it needs huge data or a strong recipe, but then it matches or beats CNNs.

14 min read982 wordscore0 viewsupdated 2026-10-11

Definition

Vision Transformer (ViT) is an image classifier that treats an image as a sequence of patches. A 224×224224\times224 image is cut into 14×14=19614\times14=196 non-overlapping 16×1616\times16 patches. Each patch is flattened and linearly projected to a DD-dimensional token, a learnable [CLS] token is prepended, learned 1-D position embeddings are added, and the 197197 tokens go through a standard Transformer encoder. The class is read from the final [CLS] token.

Dosovitskiy et al. introduced ViT in "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (ICLR 2021). The idea was to use the NLP Transformer almost unchanged, with no convolutions after the patch projection. Earlier work had mixed attention into CNNs or applied it to pixels or small local neighbourhoods. ViT showed that a pure Transformer on patches works, if it is pre-trained on enough data.

The data requirement is the main result. Trained on ImageNet-1K alone (1.3M images), ViT-B/16 reached 77.9% top-1 in the paper (Table 5, fine-tuned at 384²), below comparable ResNets. Pre-trained on ImageNet-21k (14M images) or Google's internal JFT-300M (300M images) and then fine-tuned, the models overtook the best CNNs: ViT-H/14 pre-trained on JFT-300M reached 88.55% ImageNet top-1, against 87.54% for the BiT-L ResNet, using less pre-training compute. A year later Touvron et al.'s DeiT ("Training data-efficient image transformers & distillation through attention", ICML 2021) showed that ViT-B trains to 81.8% on ImageNet-1K alone with a strong recipe (300 epochs, AdamW, RandAugment, Mixup, CutMix, random erasing, Stochastic depth, repeated augmentation), which made ViTs practical without private data.

The model

z0=[ xcls;  xp1E;  xp2E;  … ;  xpNE ]+Eposzℓ′=MSA(LN(zℓ−1))+zℓ−1zℓ=MLP(LN(zℓ′))+zℓ′ℓ=1…Ly=head(LN(zL0))\begin{aligned}z_0&=\big[\,x_{\text{cls}};\;x_p^1E;\;x_p^2E;\;\dots;\;x_p^{N}E\,\big]+E_{\text{pos}}\\ z'_\ell&=\mathrm{MSA}\big(\mathrm{LN}(z_{\ell-1})\big)+z_{\ell-1}\\ z_\ell&=\mathrm{MLP}\big(\mathrm{LN}(z'_\ell)\big)+z'_\ell\qquad \ell=1\dots L\\ y&=\mathrm{head}\big(\mathrm{LN}(z_L^0)\big)\end{aligned}
xpi∈RP2⋅3x_p^i\in\mathbb R^{P^2\cdot 3}
the ii-th flattened P×P×3P\times P\times3 patch, N=HW/P2N=HW/P^2 of them (row-major order)
E∈R(P2⋅3)×DE\in\mathbb R^{(P^2\cdot3)\times D}
the shared patch projection; in code a P×PP\times P convolution with stride PP
xclsx_{\text{cls}}
a learned DD-vector prepended to the sequence; its final state zL0z_L^0 is the image representation
Epos∈R(N+1)×DE_{\text{pos}}\in\mathbb R^{(N+1)\times D}
learned 1-D position embeddings, one per token slot
MSA\mathrm{MSA}
multi-head Self-Attention over all N+1N+1 tokens
LN\mathrm{LN}
Layer normalization, applied before each sub-layer (pre-LN), unlike the post-LN original Transformer
MLP\mathrm{MLP}
D→4D→DD\to4D\to D with GELU

Every token is normalized, attends to every other token, is normalized again and goes through the same MLP. Nothing in the encoder knows the tokens came from a 2-D grid. The only spatial information is the position embedding, and the paper found that 2-D-aware or relative position embeddings gave no clear gain over the plain learned 1-D ones: the model learns the 2-D structure itself, and nearby patches end up with similar position embeddings.

ViT-B/16 at a glance (torchvision vit_b_16)

Tokens
(224/16)2+1=197(224/16)^2+1=197 (196196 patches + [CLS])
Width / depth / heads
D=768D=768, L=12L=12, 12 heads of 64, MLP 3072
Parameters
86,567,656 (paper: 86M)
Cost
17.56 GMACs per 2242224^2 image (torchvision lists "17.56 GFLOPS")
Attention matrix
1972=38,809197^2=38{,}809 entries per head per layer
Inductive bias
only the patch grid; no locality or translation equivariance beyond it
Accuracy
81.07% acc@1 (torchvision IMAGENET1K_V1, trained on ImageNet-1K with a DeiT-like recipe)
224×224 image → 14×14 patches of 16×16×3 = 768 valuestokens, each + its learned E_pos[0…196]CLS12345678910… 196click a patch to find its tokenrow-major order, [CLS] firstViT-Base/16: D=768, L=12, heads=12, MLP=3072tokens N(224/16)² + 1 = 197parameters86,567,656cost17.56 GMACsattention matrix38,809 entries / head / layerN² share of cost4.1%where the MACs go■ patch embed 115.61M■ qkv + proj 5.58G■ QKᵀ, A·V (N²) 715.33M■ MLP 11.16GGMACs vs image size (patch 16); rose = the N² attention part17.6224px · N=19755.5384px · N=57778.5448px · N=785107.0518px · N=1025cost = L·[4ND² + 2N²D + 2N·D·D_mlp] + embed + head (MACs)
model
patch
image
fig.A procedurally drawn image is cut into patches; click a patch to find its token. The right panel counts the chosen ViT from its real configuration. For ViT-B/16 at 224 it gives 197 tokens, 86,567,656 parameters and 17.56 GMACs, of which only 4.1% is the N² part (QKᵀ and A·V): the MLPs take 11.16G and the q/k/v/output projections 5.58G. Click the bars or change the image size: at 384 the token count is 577 and the cost 55.48G (11.1% N²). Shrinking the patch to 8×8 at 224 quadruples the tokens to 785, the cost to 78.15G and the attention matrix to 616,225 entries, while the parameter count barely moves.

Where the cost goes

Per layer, the q, k, v and output projections cost 4ND24ND^2 multiply-adds, the two attention products QK⊤QK^\top and AVAV cost 2N2D2N^2D, and the MLP costs 2N⋅D⋅4D=8ND22N\cdot D\cdot4D=8ND^2. Only the middle term is quadratic in the number of tokens:

MACs≈L (12ND2+2N2D),N=(HP)2+1\text{MACs}\approx L\,\big(12ND^2+2N^2D\big),\qquad N=\Big(\frac{H}{P}\Big)^{2}+1
12ND212ND^2
projections (4ND24ND^2) plus MLP (8ND28ND^2): linear in NN
2N2D2N^2D
attention scores and weighted sum: quadratic in NN, i.e. quartic in image side length HH for fixed PP
PP
patch size; halving it multiplies N by about 4

At 2242224^2 with 16×1616\times16 patches N=197N=197 is much smaller than 6D=46086D=4608, so the MLPs dominate and the quadratic term is only 4% of the cost. The quadratic term grows fast once NN approaches DD: at 3842384^2 it is 11%, and with 8×88\times8 patches 14.5%. That is why ViTs use large patches, why dense prediction at high resolution is expensive for them, and why Swin Transformer restricts attention to local windows.

Worked example: counting ViT-B/16

componentshapeparameters
patch embeddingConv2d(3, 768, 16, stride 16): 3⋅162⋅768+7683\cdot16^2\cdot768+768590,592
[CLS] token1×7681\times768768
position embeddings197×768197\times768151,296
one encoder block2 LN + qkv 768→2304768\to2304 + proj + MLP 768→3072→768768\to3072\to7687,087,872
12 blocks12×7,087,87212\times7{,}087{,}87285,054,464
final LN + head2⋅768+768⋅1000+10002\cdot768+768\cdot1000+1000770,536
total86,567,656
Parameters of torchvision's vit_b_16 by component (computed by the figure's code; total matches torchvision's documented 86,567,656)

Multiply-adds of one encoder block at 224² (N = 197, D = 768)

  1. 1Projections q,k,vq,k,v and output: 4ND2=4⋅197⋅7682=464.84ND^2=4\cdot197\cdot768^2=464.8M.
  2. 2Attention QK⊤QK^\top and AVAV: 2N2D=2⋅1972⋅768=59.62N^2D=2\cdot197^2\cdot768=59.6M, summed over the 12 heads of width 64.
  3. 3MLP: 2⋅N⋅D⋅3072=929.62\cdot N\cdot D\cdot 3072=929.6M.
  4. 4Block total 1,454.0M; times 12 blocks = 17.45G. Adding the patch embedding (196⋅768⋅768=115.6196\cdot768\cdot768=115.6M) and the head (0.77M) gives 17.56 GMACs, the number torchvision reports.

Why it needs so much data

A CNN hard-codes two assumptions: locality (a unit sees a small neighbourhood) and translation equivariance (the same filter everywhere). They are correct for natural images, so the network does not have to learn them. In ViT only the patchification and the shared MLP carry anything similar. Self-attention is global from the first layer, and the position embeddings start random, so the model has to discover from data that nearby patches matter more.

With little data this weak prior loses: there are many functions consistent with the training set and the CNN's bias picks better ones. With a lot of data the prior stops helping and can even limit what the model represents. The ViT paper's Figure 3 shows the crossover: on ImageNet-1K, ResNets (BiT) beat every ViT; on ImageNet-21k they are about even; on JFT-300M the larger ViTs win. Its attention-distance analysis shows that some heads in the lowest layers already attend globally, while others stay local, behaving like early convolutions.

Gradients flow as in any pre-LN Transformer: each block adds its output to the residual stream, so ∂zℓ/∂zℓ−1=I+∂(MSA∘LN)/∂zℓ−1\partial z_\ell/\partial z_{\ell-1}=I+\partial(\mathrm{MSA}\circ\mathrm{LN})/\partial z_{\ell-1} and the identity term carries the gradient straight to the patch embedding, as in a Residual neural network (ResNet). Inside attention, the softmax Jacobian diag⁡(a)−aa⊤\operatorname{diag}(a)-aa^\top routes gradient mostly to the keys that received high attention weight.

python
1import torch, torch.nn as nn
2from torchvision.models import vit_b_16, ViT_B_16_Weights
3
4B, D, P = 2, 768, 16
5img = torch.randn(B, 3, 224, 224)
6
7# 1) patch embedding: a PxP conv with stride P == linear map of each flattened patch
8proj = nn.Conv2d(3, D, kernel_size=P, stride=P)
9x = proj(img) # (B, 768, 14, 14)
10x = x.flatten(2).transpose(1, 2) # (B, 196, 768), row-major patch order
11
12# 2) prepend [CLS], add learned 1-D position embeddings
13cls = nn.Parameter(torch.zeros(1, 1, D))
14pos = nn.Parameter(torch.randn(1, 197, D) * 0.02)
15z = torch.cat([cls.expand(B, -1, -1), x], dim=1) + pos # (B, 197, 768)
16
17# 3) pre-LN encoder (norm_first=True), GELU MLP of width 4D
18layer = nn.TransformerEncoderLayer(D, nhead=12, dim_feedforward=4 * D, activation="gelu",
19 batch_first=True, norm_first=True, dropout=0.0)
20enc = nn.TransformerEncoder(layer, num_layers=12)
21logits = nn.Linear(D, 1000)(nn.LayerNorm(D)(enc(z)[:, 0])) # read the [CLS] token
22print(logits.shape) # torch.Size([2, 1000])
23
24# The reference model
25model = vit_b_16(weights=ViT_B_16_Weights.IMAGENET1K_V1).eval()
26print(sum(p.numel() for p in model.parameters())) # 86567656
27print(model.conv_proj) # Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
28print(model.encoder.pos_embedding.shape) # torch.Size([1, 197, 768])
Patchify + [CLS] + position embeddings by hand, then the torchvision model

Use when

  • +Simple and uniform: one block type, no image-specific engineering beyond the patch grid
  • +Global receptive field from the first layer
  • +Scales very well with data and model size; the same architecture as language models, so the same tooling and hardware kernels
  • +Strong backbone for self-supervised (MAE, DINO) and multimodal (CLIP) pre-training

Avoid when

  • −Weak inductive bias: poor results from scratch on small datasets without a heavy augmentation/regularization recipe
  • −Attention cost grows as N2N^2, i.e. as (H/P)4(H/P)^4 in the image side; costly for high resolution and small patches
  • −Single-scale 16×16\times downsampled tokens: no feature pyramid for detection without extra adapters
  • −Position embeddings tied to the training grid; must be interpolated for new resolutions

Check yourself

A ViT-B with 16×16 patches is applied to a 384×384 image. How many tokens enter the encoder?

Check yourself

Trained on ImageNet-1K alone with a basic recipe, ViT-B/16 underperforms a ResNet of similar size. What is the main reason given by the ViT paper?

Remember

  • ✓ViT = patchify (P×PP\times P conv, stride PP) + [CLS] + learned position embeddings + a standard pre-LN Transformer encoder.
  • ✓ViT-B/16 at 224: 197 tokens, 86.6M parameters, 17.56 GMACs.
  • ✓Cost per layer ≈12ND2+2N2D\approx12ND^2+2N^2D: MLPs dominate at 224, but the N2N^2 term grows quartically with resolution.
  • ✓Weak inductive bias: it needs ImageNet-21k / JFT-300M pre-training or DeiT's recipe to beat CNNs.
  • ✓New resolutions need interpolated position embeddings; Swin and ConvNeXt were answers to ViT's cost and missing hierarchy.
← back to Deep Learning roadmap