Image classification

Map an image to one of K labels: a backbone, global pooling, a linear layer and a softmax trained with cross-entropy. ImageNet-1k (1.28M train / 50k val, 1000 classes) is the benchmark, scored by top-1 and top-5 accuracy.

7 min read731 wordscore0 viewsupdated 2026-10-15

Definition

Image classification is the task of assigning a whole image one label from a fixed set of KK classes. The model outputs a probability distribution p∈ΔK−1p\in\Delta^{K-1}, and the prediction is arg⁡max⁡kpk\arg\max_k p_k. The reference benchmark is the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 subset of ImageNet (Deng et al., CVPR 2009), now usually called ImageNet-1k.

Input: an RGB tensor, usually 3×224×2243\times224\times224 after resizing and cropping. Output: KK scores. The task is hard because one label covers huge variation in pose, scale, lighting, occlusion and background, and some classes differ in fine detail, such as ImageNet's many dog breeds. Before 2012 the best systems used hand-built features (SIFT, HOG, Fisher vectors) and a linear SVM. AlexNet cut ILSVRC-2012 top-5 error from 26.2% to 15.3% and ended that era.

The standard recipe

Every modern classifier has the same layout. A backbone turns the image into a feature map, global average pooling (GAP) turns the map into one vector hh, and a linear layer gives logits z=Wh+bz=Wh+b. Softmax turns the logits into probabilities. Training minimises the cross-entropy of the true label yy:

pk=ezk/T∑j=1Kezj/T,L=−log⁡py,∂L∂zk=pk−1[k=y](T=1)p_k=\frac{e^{z_k/T}}{\sum_{j=1}^{K}e^{z_j/T}},\qquad \mathcal L=-\log p_y,\qquad \frac{\partial\mathcal L}{\partial z_k}=p_k-\mathbb 1[k=y]\quad(T=1)
z∈RKz\in\mathbb R^K
logits from the linear head, K=1000K=1000 for ImageNet-1k
TT
temperature; T=1T=1 in training, tuned after training for calibration
yy
index of the true class
image → backbone → pooled feature → linear head → softmax over K = 1000 ImageNet classes3×224×224normalisedbackbone: ResNet-5056×56×256/428×28×512/814×14×1024/167×7×2048/32GAP2048FC2048→1000 zsoftmax p: top-5tabby cat 0.62tiger cat 0.21Egyptian cat 0.09lynx 0.03Persian cat 0.01(illustrative scores)trainloss = −log p_y(cross-entropy, label y)ImageNet-1k: 1.28M trainimages, 50k valaug: random crop, flipevaluatetop-1: arg-max = ytop-5: y in 5 highesthere: top-1 ✓ (tabby)calibration: is p = 0.62right 62% of the time?transfer to a new taskkeep pretrained backboneswap FC 2048×1000 fora new 2048×K headlinear probe: freeze itfine-tune: train all, low lr
fig.A ResNet-50 classifier. Four stages shrink the 224 × 224 input to a 7 × 7 × 2048 map (stride 32), global average pooling gives a 2048-vector, and a 2048 × 1000 linear layer gives the logits. For transfer learning only the head is swapped; the scores shown are illustrative.

The backbone is where architectures differ. The arc runs from LeNet-5 (digits, 1998) to AlexNet (2012), VGGNet and GoogLeNet / Inception (2014), ResNet (2015, residual blocks that make 50 to 152 layers trainable), EfficientNet (compound scaling) and ConvNeXt among CNNs. Vision Transformers (ViT) split the image into 16×1616\times16 patches and replace convolution with self-attention; Swin Transformer brings back locality and a feature pyramid. The head and the loss stay the same throughout.

Benchmark and reference numbers

ImageNet-1k
1000 classes; 1,281,167 training images, 50,000 validation (50 per class), 100,000 test; results are normally reported on the validation set
Top-1 / top-5
top-1: the arg-max is the true label; top-5: the true label is among the 5 highest scores. Top-5 was the ILSVRC metric because many images contain several objects
ILSVRC winners (top-5 error)
AlexNet 15.3% (2012); ResNet ensemble 3.57% (2015)
ResNet-50
25.6M parameters, 4.1G multiply-adds; torchvision weights: 76.13% top-1 (V1) and 80.86% (V2, same architecture, newer training recipe)
Evaluation crop
classic protocol: resize the short side to 256, centre-crop 224×224224\times224, normalise with the ImageNet mean (0.485, 0.456, 0.406) and std (0.229, 0.224, 0.225)

Augmentation and transfer learning

The default training augmentation is the Inception-style random-resized crop (a random region covering 8% to 100% of the image, at a random aspect ratio, resized to 224) plus a horizontal flip. Modern recipes add RandAugment, Mixup, CutMix, Label smoothing, Stochastic depth and long schedules. These recipe changes alone account for the 4.7-point gap between the two ResNet-50 checkpoints above (see Data augmentation).

Features learned on ImageNet transfer. To solve a new KK-class task, load the pretrained backbone and replace the 2048×10002048\times1000 head with a fresh 2048×K2048\times K layer. A linear probe freezes the backbone and trains only the head; fine-tuning trains everything with a learning rate well below the pretraining one. Fine-tuning usually wins when there are thousands of labelled images; a linear probe is cheaper and safer with very few.

Calibration

A classifier is calibrated if, among the images it predicts with confidence 0.8, 80% are correct. Guo, Pleiss, Sun and Weinberger ("On Calibration of Modern Neural Networks", ICML 2017) showed that deep, heavily trained networks such as ResNets are over-confident. They measured it with the expected calibration error over MM confidence bins:

ECE=∑m=1M∣Bm∣n ∣acc⁡(Bm)−conf⁡(Bm)∣\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\,\big|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\big|
BmB_m
validation images whose top probability max⁡kpk\max_k p_k falls in bin mm (e.g. M=10M=10 or 15 equal-width bins)
acc⁡, conf⁡\operatorname{acc},\ \operatorname{conf}
fraction correct and mean top probability inside the bin

Their fix is temperature scaling: learn one scalar T>0T>0 on the validation set by minimising the negative log-likelihood (NLL), and divide all logits by it. Dividing by a positive constant does not change the order of the logits, so top-1 and top-5 accuracy are unchanged; only the confidences move.

softmax(z / T) on 600 validation images, 10 classes · T = 1.00reliability: accuracy per confidence bin000.50.511confidence max_k p_k (darker bar = more images)rose = over-confident gapmint = under-confident gapone image (label: cat), p = softmax(z / T)airplane0.000car0.000bird0.000cat0.002deer0.000dog0.993frog0.000horse0.001ship0.000truck0.003tinted rows = top-5 · top-1 · true classtrue class rank 3: top-1 ✗, top-5 ✓top-1 73.7% · top-5 94.7% · mean confidence 96.4%NLL 1.662 (T=1: 1.662) · ECE 0.227 (T=1: 0.227)dividing by T > 0 keeps the arg-max and the ranking, so top-1 and top-5 never move; NLL-optimal T = 2.55
temperature
example
fig.A synthetic 600-image, 10-class validation set whose logits are 2.5× too large. At T = 1, top-1 is 73.7%, top-5 94.7% and mean confidence 96.4%: NLL 1.662, ECE 0.227, and the reliability bars sit far below the diagonal. "Fit T" finds T = 2.55: confidence falls to 74.8%, NLL to 0.964 and ECE to 0.034, while top-1 and top-5 do not move. At T = 4 the model becomes under-confident (confidence 53.2%, ECE 0.205). The "cat → dog" image is 99.3% dog at T = 1 and 70.7% at T = 2.55; cat stays rank 3, a top-5 hit.

Worked example: one image, five classes

Logits z=(4.0, 2.5, 1.0, 0.5, −1.0)z=(4.0,\ 2.5,\ 1.0,\ 0.5,\ -1.0) for (tabby cat, tiger cat, lynx, fox, car); the true label is tiger cat.

Classzp at T = 1p at T = 2
tabby cat4.00.76340.5125
tiger cat (true)2.50.17030.2421
lynx1.00.03800.1143
fox0.50.02310.0891
car−1.00.00510.0421
loss −log p_y1.76991.4185
Softmax at T = 1 and T = 2 (sums of exponentials 71.5155 and 14.4187)

Top-1 is wrong (tabby cat), top-5 is right (tiger cat is rank 2), at both temperatures. T=2T=2 keeps the ranking but lowers the loss from 1.770 to 1.419, because the over-confident 0.763 on the wrong class drops to 0.513. Averaged over a validation set, that NLL reduction is what temperature scaling optimises.

python
1import torch, torch.nn as nn
2from torchvision import models, transforms
3
4weights = models.ResNet50_Weights.IMAGENET1K_V2
5model = models.resnet50(weights=weights)
6model.fc = nn.Linear(model.fc.in_features, 10) # 2048 -> 10, new head
7
8train_tf = transforms.Compose([
9 transforms.RandomResizedCrop(224), # scale (0.08, 1.0) by default
10 transforms.RandomHorizontalFlip(),
11 transforms.ToTensor(),
12 transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
13])
14eval_tf = weights.transforms() # resize, centre-crop 224, normalise
15
16# backbone at a low rate, fresh head at a higher one
17opt = torch.optim.SGD([
18 {"params": [p for n, p in model.named_parameters() if not n.startswith("fc.")], "lr": 1e-3},
19 {"params": model.fc.parameters(), "lr": 1e-2},
20], momentum=0.9, weight_decay=1e-4)
21loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)
22
23@torch.no_grad()
24def topk(logits, y, ks=(1, 5)):
25 top = logits.topk(max(ks), dim=1).indices # (B, 5)
26 hit = top == y[:, None]
27 return [hit[:, :k].any(1).float().mean().item() for k in ks]
28
29def fit_temperature(val_logits, val_y):
30 """Temperature scaling (Guo et al., 2017): one scalar T, minimise validation NLL."""
31 log_t = torch.zeros(1, requires_grad=True)
32 opt_t = torch.optim.LBFGS([log_t], lr=0.1, max_iter=100)
33 def closure():
34 opt_t.zero_grad()
35 loss = nn.functional.cross_entropy(val_logits / log_t.exp(), val_y)
36 loss.backward()
37 return loss
38 opt_t.step(closure)
39 return log_t.exp().item()
Fine-tuning a pretrained ResNet-50 on a new 10-class task with torchvision (not run here: needs torch, torchvision and a dataset)

Check yourself

After temperature scaling with T = 2.5, what happens to top-1 and top-5 accuracy?

Check yourself

You have 300 labelled images for a new 5-class task and an ImageNet-pretrained ResNet-50. What is the safest first baseline?

Remember

  • ✓Classifier = backbone → global average pooling → linear layer → softmax, trained with cross-entropy; the gradient on the logits is p − onehot(y).
  • ✓ImageNet-1k: 1000 classes, 1.28M training and 50k validation images, reported as top-1 and top-5 accuracy.
  • ✓Backbones changed (AlexNet, VGG, ResNet, EfficientNet, ViT, ConvNeXt); the head and loss did not.
  • ✓Transfer: swap the head for a K-way layer; linear probe (frozen) or fine-tune (all layers, low learning rate).
  • ✓Temperature scaling divides logits by one scalar T fitted on validation NLL: ECE drops, accuracy does not change.
← back to Deep Learning roadmap