# Vision Transformers (ViT)

> Cut the image into 16×16 patches, embed each as a token, add a [CLS] token and position embeddings, and run a plain Transformer encoder. Weak image priors mean it needs huge data or a strong recipe, but then it matches or beats CNNs.

Source: https://www.mlfoundry.in/learn/vision-transformers-vit  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 14 min read · updated 2026-10-11

**Vision Transformer (ViT):** is an image classifier that treats an image as a **sequence of patches**. A $224\times224$ image is cut into $14\times14=196$ non-overlapping $16\times16$ patches. Each patch is flattened and linearly projected to a $D$-dimensional token, a learnable **[CLS]** token is prepended, learned 1-D **position embeddings** are added, and the $197$ tokens go through a standard Transformer encoder. The class is read from the final [CLS] token.

Dosovitskiy et al. introduced ViT in "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (ICLR 2021). The idea was to use the NLP Transformer almost unchanged, with no convolutions after the patch projection. Earlier work had mixed attention into CNNs or applied it to pixels or small local neighbourhoods. ViT showed that a pure Transformer on patches works, **if it is pre-trained on enough data**.

The data requirement is the main result. Trained on ImageNet-1K alone (1.3M images), ViT-B/16 reached 77.9% top-1 in the paper (Table 5, fine-tuned at 384²), below comparable ResNets. Pre-trained on ImageNet-21k (14M images) or Google's internal JFT-300M (300M images) and then fine-tuned, the models overtook the best CNNs: ViT-H/14 pre-trained on JFT-300M reached 88.55% ImageNet top-1, against 87.54% for the BiT-L ResNet, using less pre-training compute. A year later Touvron et al.'s **DeiT** ("Training data-efficient image transformers & distillation through attention", ICML 2021) showed that ViT-B trains to 81.8% on ImageNet-1K alone with a strong recipe (300 epochs, **AdamW**, RandAugment, **Mixup**, **CutMix**, random erasing, **Stochastic depth**, repeated augmentation), which made ViTs practical without private data.

## The model

$$
\begin{aligned}z_0&=\big[\,x_{\text{cls}};\;x_p^1E;\;x_p^2E;\;\dots;\;x_p^{N}E\,\big]+E_{\text{pos}}\\ z'_\ell&=\mathrm{MSA}\big(\mathrm{LN}(z_{\ell-1})\big)+z_{\ell-1}\\ z_\ell&=\mathrm{MLP}\big(\mathrm{LN}(z'_\ell)\big)+z'_\ell\qquad \ell=1\dots L\\ y&=\mathrm{head}\big(\mathrm{LN}(z_L^0)\big)\end{aligned}
$$

- $x_p^i\in\mathbb R^{P^2\cdot 3}$: the $i$-th flattened $P\times P\times3$ patch, $N=HW/P^2$ of them (row-major order)
- $E\in\mathbb R^{(P^2\cdot3)\times D}$: the shared patch projection; in code a $P\times P$ convolution with stride $P$
- $x_{\text{cls}}$: a learned $D$-vector prepended to the sequence; its final state $z_L^0$ is the image representation
- $E_{\text{pos}}\in\mathbb R^{(N+1)\times D}$: learned 1-D position embeddings, one per token slot
- $\mathrm{MSA}$: multi-head **Self-Attention** over all $N+1$ tokens
- $\mathrm{LN}$: **Layer normalization**, applied *before* each sub-layer (pre-LN), unlike the post-LN original Transformer
- $\mathrm{MLP}$: $D\to4D\to D$ with GELU

Every token is normalized, attends to every other token, is normalized again and goes through the same MLP. Nothing in the encoder knows the tokens came from a 2-D grid. The only spatial information is the position embedding, and the paper found that 2-D-aware or relative position embeddings gave no clear gain over the plain learned 1-D ones: the model learns the 2-D structure itself, and nearby patches end up with similar position embeddings.

**ViT-B/16 at a glance (torchvision vit_b_16)**

- **Tokens:** $(224/16)^2+1=197$ ($196$ patches + [CLS])
- **Width / depth / heads:** $D=768$, $L=12$, 12 heads of 64, MLP 3072
- **Parameters:** 86,567,656 (paper: 86M)
- **Cost:** 17.56 GMACs per $224^2$ image (torchvision lists "17.56 GFLOPS")
- **Attention matrix:** $197^2=38{,}809$ entries per head per layer
- **Inductive bias:** only the patch grid; no locality or translation equivariance beyond it
- **Accuracy:** 81.07% acc@1 (torchvision IMAGENET1K_V1, trained on ImageNet-1K with a DeiT-like recipe)

> Figure: A procedurally drawn image is cut into patches; click a patch to find its token. The right panel counts the chosen ViT from its real configuration. For ViT-B/16 at 224 it gives 197 tokens, 86,567,656 parameters and 17.56 GMACs, of which only 4.1% is the N² part (QKᵀ and A·V): the MLPs take 11.16G and the q/k/v/output projections 5.58G. Click the bars or change the image size: at 384 the token count is 577 and the cost 55.48G (11.1% N²). Shrinking the patch to 8×8 at 224 quadruples the tokens to 785, the cost to 78.15G and the attention matrix to 616,225 entries, while the parameter count barely moves.

## Where the cost goes

Per layer, the q, k, v and output projections cost $4ND^2$ multiply-adds, the two attention products $QK^\top$ and $AV$ cost $2N^2D$, and the MLP costs $2N\cdot D\cdot4D=8ND^2$. Only the middle term is quadratic in the number of tokens:

$$
\text{MACs}\approx L\,\big(12ND^2+2N^2D\big),\qquad N=\Big(\frac{H}{P}\Big)^{2}+1
$$

- $12ND^2$: projections ($4ND^2$) plus MLP ($8ND^2$): linear in $N$
- $2N^2D$: attention scores and weighted sum: quadratic in $N$, i.e. **quartic** in image side length $H$ for fixed $P$
- $P$: patch size; halving it multiplies N by about 4

At $224^2$ with $16\times16$ patches $N=197$ is much smaller than $6D=4608$, so the MLPs dominate and the quadratic term is only 4% of the cost. The quadratic term grows fast once $N$ approaches $D$: at $384^2$ it is 11%, and with $8\times8$ patches 14.5%. That is why ViTs use large patches, why dense prediction at high resolution is expensive for them, and why **Swin Transformer** restricts attention to local windows.

## Worked example: counting ViT-B/16

Parameters of torchvision's vit_b_16 by component (computed by the figure's code; total matches torchvision's documented 86,567,656)

| component | shape | parameters |
| --- | --- | --- |
| patch embedding | Conv2d(3, 768, 16, stride 16): $3\cdot16^2\cdot768+768$ | 590,592 |
| [CLS] token | $1\times768$ | 768 |
| position embeddings | $197\times768$ | 151,296 |
| one encoder block | 2 LN + qkv $768\to2304$ + proj + MLP $768\to3072\to768$ | 7,087,872 |
| 12 blocks | $12\times7{,}087{,}872$ | 85,054,464 |
| final LN + head | $2\cdot768+768\cdot1000+1000$ | 770,536 |
| total |  | 86,567,656 |

**Multiply-adds of one encoder block at 224² (N = 197, D = 768)**

1. Projections $q,k,v$ and output: $4ND^2=4\cdot197\cdot768^2=464.8$M.
2. Attention $QK^\top$ and $AV$: $2N^2D=2\cdot197^2\cdot768=59.6$M, summed over the 12 heads of width 64.
3. MLP: $2\cdot N\cdot D\cdot 3072=929.6$M.
4. Block total 1,454.0M; times 12 blocks = 17.45G. Adding the patch embedding ($196\cdot768\cdot768=115.6$M) and the head (0.77M) gives **17.56 GMACs**, the number torchvision reports.

## Why it needs so much data

A CNN hard-codes two assumptions: **locality** (a unit sees a small neighbourhood) and **translation equivariance** (the same filter everywhere). They are correct for natural images, so the network does not have to learn them. In ViT only the patchification and the shared MLP carry anything similar. Self-attention is global from the first layer, and the position embeddings start random, so the model has to discover from data that nearby patches matter more.

With little data this weak prior loses: there are many functions consistent with the training set and the CNN's bias picks better ones. With a lot of data the prior stops helping and can even limit what the model represents. The ViT paper's Figure 3 shows the crossover: on ImageNet-1K, ResNets (BiT) beat every ViT; on ImageNet-21k they are about even; on JFT-300M the larger ViTs win. Its attention-distance analysis shows that some heads in the lowest layers already attend globally, while others stay local, behaving like early convolutions.

Gradients flow as in any pre-LN Transformer: each block adds its output to the residual stream, so $\partial z_\ell/\partial z_{\ell-1}=I+\partial(\mathrm{MSA}\circ\mathrm{LN})/\partial z_{\ell-1}$ and the identity term carries the gradient straight to the patch embedding, as in a **Residual neural network (ResNet)**. Inside attention, the softmax Jacobian $\operatorname{diag}(a)-aa^\top$ routes gradient mostly to the keys that received high attention weight.

```python
import torch, torch.nn as nn
from torchvision.models import vit_b_16, ViT_B_16_Weights

B, D, P = 2, 768, 16
img = torch.randn(B, 3, 224, 224)

# 1) patch embedding: a PxP conv with stride P == linear map of each flattened patch
proj = nn.Conv2d(3, D, kernel_size=P, stride=P)
x = proj(img)                       # (B, 768, 14, 14)
x = x.flatten(2).transpose(1, 2)    # (B, 196, 768), row-major patch order

# 2) prepend [CLS], add learned 1-D position embeddings
cls = nn.Parameter(torch.zeros(1, 1, D))
pos = nn.Parameter(torch.randn(1, 197, D) * 0.02)
z = torch.cat([cls.expand(B, -1, -1), x], dim=1) + pos   # (B, 197, 768)

# 3) pre-LN encoder (norm_first=True), GELU MLP of width 4D
layer = nn.TransformerEncoderLayer(D, nhead=12, dim_feedforward=4 * D, activation="gelu",
                                   batch_first=True, norm_first=True, dropout=0.0)
enc = nn.TransformerEncoder(layer, num_layers=12)
logits = nn.Linear(D, 1000)(nn.LayerNorm(D)(enc(z)[:, 0]))   # read the [CLS] token
print(logits.shape)                 # torch.Size([2, 1000])

# The reference model
model = vit_b_16(weights=ViT_B_16_Weights.IMAGENET1K_V1).eval()
print(sum(p.numel() for p in model.parameters()))   # 86567656
print(model.conv_proj)              # Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
print(model.encoder.pos_embedding.shape)            # torch.Size([1, 197, 768])
```

Patchify + [CLS] + position embeddings by hand, then the torchvision model

> **Changing the resolution breaks the position embeddings:** Fine-tuning at $384^2$ with the same patch size gives $24^2=576$ patches, but the model only has 196 learned patch positions. The ViT paper **interpolates** the $14\times14$ grid of position embeddings bilinearly to $24\times24$ (keeping the [CLS] slot). This is the one place where the 2-D structure is put in by hand. Forgetting it, or feeding a size that is not a multiple of $P$, is a common bug: torchvision's `vit_b_16` asserts the input is exactly the configured `image_size`.

> **The patch embedding is a convolution:** A linear map applied to each non-overlapping $P\times P$ patch is exactly a convolution with kernel $P$ and stride $P$. So ViT has one convolution, and it is the patchify stem that **ConvNeXt** later borrowed (with $P=4$). Xiao et al. (2021, "Early Convolutions Help Transformers See Better") found that replacing it with a few small stride-2 convolutions (a "convolutional stem") makes ViT training more stable and less sensitive to the optimizer.

> **[CLS] or average pooling:** The [CLS] token comes from BERT. The ViT paper reports that global average pooling of the patch tokens works equally well if the learning rate is adjusted. Many later models (Swin, DeiT III variants, most self-supervised ViTs used as backbones) pool instead, and masked autoencoders (MAE) train ViTs by reconstructing masked patches, which is far more data-efficient than supervised pre-training.

**Strengths**

- Simple and uniform: one block type, no image-specific engineering beyond the patch grid
- Global receptive field from the first layer
- Scales very well with data and model size; the same architecture as language models, so the same tooling and hardware kernels
- Strong backbone for self-supervised (MAE, DINO) and multimodal (CLIP) pre-training

**Limitations**

- Weak inductive bias: poor results from scratch on small datasets without a heavy augmentation/regularization recipe
- Attention cost grows as $N^2$, i.e. as $(H/P)^4$ in the image side; costly for high resolution and small patches
- Single-scale $16\times$ downsampled tokens: no feature pyramid for detection without extra adapters
- Position embeddings tied to the training grid; must be interpolated for new resolutions

**Question:** A ViT-B with 16×16 patches is applied to a 384×384 image. How many tokens enter the encoder?

- A. 196
- B. 577
- C. 576
- D. 1025

**Answer:** B. 577 — $384/16=24$ patches per side, $24^2=576$ patches, plus the [CLS] token: 577. The attention matrix grows from $197^2$ to $577^2$, about 8.6×, and the whole model's cost from 17.56 to 55.48 GMACs.

**Question:** Trained on ImageNet-1K alone with a basic recipe, ViT-B/16 underperforms a ResNet of similar size. What is the main reason given by the ViT paper?

- A. Self-attention cannot represent local features
- B. The Transformer lacks the CNN's built-in locality and translation equivariance, so it needs more data to learn them
- C. The [CLS] token is a poor classifier
- D. 16×16 patches discard too much information

**Answer:** B. The Transformer lacks the CNN's built-in locality and translation equivariance, so it needs more data to learn them — Attention can learn local patterns (some heads do), but nothing forces it to. With 1.3M images the CNN's correct prior wins; with 14M–300M images, or with DeiT's heavy augmentation and regularization, the ViT catches up and then overtakes.

**Key points**

- ViT = patchify ($P\times P$ conv, stride $P$) + [CLS] + learned position embeddings + a standard pre-LN Transformer encoder.
- ViT-B/16 at 224: 197 tokens, 86.6M parameters, 17.56 GMACs.
- Cost per layer $\approx12ND^2+2N^2D$: MLPs dominate at 224, but the $N^2$ term grows quartically with resolution.
- Weak inductive bias: it needs ImageNet-21k / JFT-300M pre-training or DeiT's recipe to beat CNNs.
- New resolutions need interpolated position embeddings; Swin and ConvNeXt were answers to ViT's cost and missing hierarchy.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
