Definition
Vision Transformer (ViT) is an image classifier that treats an image as a sequence of patches. A image is cut into non-overlapping patches. Each patch is flattened and linearly projected to a -dimensional token, a learnable [CLS] token is prepended, learned 1-D position embeddings are added, and the tokens go through a standard Transformer encoder. The class is read from the final [CLS] token.
Dosovitskiy et al. introduced ViT in "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (ICLR 2021). The idea was to use the NLP Transformer almost unchanged, with no convolutions after the patch projection. Earlier work had mixed attention into CNNs or applied it to pixels or small local neighbourhoods. ViT showed that a pure Transformer on patches works, if it is pre-trained on enough data.
The data requirement is the main result. Trained on ImageNet-1K alone (1.3M images), ViT-B/16 reached 77.9% top-1 in the paper (Table 5, fine-tuned at 384²), below comparable ResNets. Pre-trained on ImageNet-21k (14M images) or Google's internal JFT-300M (300M images) and then fine-tuned, the models overtook the best CNNs: ViT-H/14 pre-trained on JFT-300M reached 88.55% ImageNet top-1, against 87.54% for the BiT-L ResNet, using less pre-training compute. A year later Touvron et al.'s DeiT ("Training data-efficient image transformers & distillation through attention", ICML 2021) showed that ViT-B trains to 81.8% on ImageNet-1K alone with a strong recipe (300 epochs, AdamW, RandAugment, Mixup, CutMix, random erasing, Stochastic depth, repeated augmentation), which made ViTs practical without private data.
The model
- the -th flattened patch, of them (row-major order)
- the shared patch projection; in code a convolution with stride
- a learned -vector prepended to the sequence; its final state is the image representation
- learned 1-D position embeddings, one per token slot
- multi-head Self-Attention over all tokens
- Layer normalization, applied before each sub-layer (pre-LN), unlike the post-LN original Transformer
- with GELU
Every token is normalized, attends to every other token, is normalized again and goes through the same MLP. Nothing in the encoder knows the tokens came from a 2-D grid. The only spatial information is the position embedding, and the paper found that 2-D-aware or relative position embeddings gave no clear gain over the plain learned 1-D ones: the model learns the 2-D structure itself, and nearby patches end up with similar position embeddings.
ViT-B/16 at a glance (torchvision vit_b_16)
- Tokens
- ( patches + [CLS])
- Width / depth / heads
- , , 12 heads of 64, MLP 3072
- Parameters
- 86,567,656 (paper: 86M)
- Cost
- 17.56 GMACs per image (torchvision lists "17.56 GFLOPS")
- Attention matrix
- entries per head per layer
- Inductive bias
- only the patch grid; no locality or translation equivariance beyond it
- Accuracy
- 81.07% acc@1 (torchvision IMAGENET1K_V1, trained on ImageNet-1K with a DeiT-like recipe)
Where the cost goes
Per layer, the q, k, v and output projections cost multiply-adds, the two attention products and cost , and the MLP costs . Only the middle term is quadratic in the number of tokens:
- projections () plus MLP (): linear in
- attention scores and weighted sum: quadratic in , i.e. quartic in image side length for fixed
- patch size; halving it multiplies N by about 4
At with patches is much smaller than , so the MLPs dominate and the quadratic term is only 4% of the cost. The quadratic term grows fast once approaches : at it is 11%, and with patches 14.5%. That is why ViTs use large patches, why dense prediction at high resolution is expensive for them, and why Swin Transformer restricts attention to local windows.
Worked example: counting ViT-B/16
| component | shape | parameters |
|---|---|---|
| patch embedding | Conv2d(3, 768, 16, stride 16): | 590,592 |
| [CLS] token | 768 | |
| position embeddings | 151,296 | |
| one encoder block | 2 LN + qkv + proj + MLP | 7,087,872 |
| 12 blocks | 85,054,464 | |
| final LN + head | 770,536 | |
| total | 86,567,656 |
Multiply-adds of one encoder block at 224² (N = 197, D = 768)
- 1Projections and output: M.
- 2Attention and : M, summed over the 12 heads of width 64.
- 3MLP: M.
- 4Block total 1,454.0M; times 12 blocks = 17.45G. Adding the patch embedding (M) and the head (0.77M) gives 17.56 GMACs, the number torchvision reports.
Why it needs so much data
A CNN hard-codes two assumptions: locality (a unit sees a small neighbourhood) and translation equivariance (the same filter everywhere). They are correct for natural images, so the network does not have to learn them. In ViT only the patchification and the shared MLP carry anything similar. Self-attention is global from the first layer, and the position embeddings start random, so the model has to discover from data that nearby patches matter more.
With little data this weak prior loses: there are many functions consistent with the training set and the CNN's bias picks better ones. With a lot of data the prior stops helping and can even limit what the model represents. The ViT paper's Figure 3 shows the crossover: on ImageNet-1K, ResNets (BiT) beat every ViT; on ImageNet-21k they are about even; on JFT-300M the larger ViTs win. Its attention-distance analysis shows that some heads in the lowest layers already attend globally, while others stay local, behaving like early convolutions.
Gradients flow as in any pre-LN Transformer: each block adds its output to the residual stream, so and the identity term carries the gradient straight to the patch embedding, as in a Residual neural network (ResNet). Inside attention, the softmax Jacobian routes gradient mostly to the keys that received high attention weight.
1import torch, torch.nn as nn2from torchvision.models import vit_b_16, ViT_B_16_Weights34B, D, P = 2, 768, 165img = torch.randn(B, 3, 224, 224)67# 1) patch embedding: a PxP conv with stride P == linear map of each flattened patch8proj = nn.Conv2d(3, D, kernel_size=P, stride=P)9x = proj(img) # (B, 768, 14, 14)10x = x.flatten(2).transpose(1, 2) # (B, 196, 768), row-major patch order1112# 2) prepend [CLS], add learned 1-D position embeddings13cls = nn.Parameter(torch.zeros(1, 1, D))14pos = nn.Parameter(torch.randn(1, 197, D) * 0.02)15z = torch.cat([cls.expand(B, -1, -1), x], dim=1) + pos # (B, 197, 768)1617# 3) pre-LN encoder (norm_first=True), GELU MLP of width 4D18layer = nn.TransformerEncoderLayer(D, nhead=12, dim_feedforward=4 * D, activation="gelu",19 batch_first=True, norm_first=True, dropout=0.0)20enc = nn.TransformerEncoder(layer, num_layers=12)21logits = nn.Linear(D, 1000)(nn.LayerNorm(D)(enc(z)[:, 0])) # read the [CLS] token22print(logits.shape) # torch.Size([2, 1000])2324# The reference model25model = vit_b_16(weights=ViT_B_16_Weights.IMAGENET1K_V1).eval()26print(sum(p.numel() for p in model.parameters())) # 8656765627print(model.conv_proj) # Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))28print(model.encoder.pos_embedding.shape) # torch.Size([1, 197, 768])Use when
- +Simple and uniform: one block type, no image-specific engineering beyond the patch grid
- +Global receptive field from the first layer
- +Scales very well with data and model size; the same architecture as language models, so the same tooling and hardware kernels
- +Strong backbone for self-supervised (MAE, DINO) and multimodal (CLIP) pre-training
Avoid when
- −Weak inductive bias: poor results from scratch on small datasets without a heavy augmentation/regularization recipe
- −Attention cost grows as , i.e. as in the image side; costly for high resolution and small patches
- −Single-scale downsampled tokens: no feature pyramid for detection without extra adapters
- −Position embeddings tied to the training grid; must be interpolated for new resolutions
Check yourself
A ViT-B with 16×16 patches is applied to a 384×384 image. How many tokens enter the encoder?
Check yourself
Trained on ImageNet-1K alone with a basic recipe, ViT-B/16 underperforms a ResNet of similar size. What is the main reason given by the ViT paper?
Remember
- ✓ViT = patchify ( conv, stride ) + [CLS] + learned position embeddings + a standard pre-LN Transformer encoder.
- ✓ViT-B/16 at 224: 197 tokens, 86.6M parameters, 17.56 GMACs.
- ✓Cost per layer : MLPs dominate at 224, but the term grows quartically with resolution.
- ✓Weak inductive bias: it needs ImageNet-21k / JFT-300M pre-training or DeiT's recipe to beat CNNs.
- ✓New resolutions need interpolated position embeddings; Swin and ConvNeXt were answers to ViT's cost and missing hierarchy.