Definition
Image classification is the task of assigning a whole image one label from a fixed set of classes. The model outputs a probability distribution , and the prediction is . The reference benchmark is the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 subset of ImageNet (Deng et al., CVPR 2009), now usually called ImageNet-1k.
Input: an RGB tensor, usually after resizing and cropping. Output: scores. The task is hard because one label covers huge variation in pose, scale, lighting, occlusion and background, and some classes differ in fine detail, such as ImageNet's many dog breeds. Before 2012 the best systems used hand-built features (SIFT, HOG, Fisher vectors) and a linear SVM. AlexNet cut ILSVRC-2012 top-5 error from 26.2% to 15.3% and ended that era.
The standard recipe
Every modern classifier has the same layout. A backbone turns the image into a feature map, global average pooling (GAP) turns the map into one vector , and a linear layer gives logits . Softmax turns the logits into probabilities. Training minimises the cross-entropy of the true label :
- logits from the linear head, for ImageNet-1k
- temperature; in training, tuned after training for calibration
- index of the true class
The backbone is where architectures differ. The arc runs from LeNet-5 (digits, 1998) to AlexNet (2012), VGGNet and GoogLeNet / Inception (2014), ResNet (2015, residual blocks that make 50 to 152 layers trainable), EfficientNet (compound scaling) and ConvNeXt among CNNs. Vision Transformers (ViT) split the image into patches and replace convolution with self-attention; Swin Transformer brings back locality and a feature pyramid. The head and the loss stay the same throughout.
Benchmark and reference numbers
- ImageNet-1k
- 1000 classes; 1,281,167 training images, 50,000 validation (50 per class), 100,000 test; results are normally reported on the validation set
- Top-1 / top-5
- top-1: the arg-max is the true label; top-5: the true label is among the 5 highest scores. Top-5 was the ILSVRC metric because many images contain several objects
- ILSVRC winners (top-5 error)
- AlexNet 15.3% (2012); ResNet ensemble 3.57% (2015)
- ResNet-50
- 25.6M parameters, 4.1G multiply-adds; torchvision weights: 76.13% top-1 (V1) and 80.86% (V2, same architecture, newer training recipe)
- Evaluation crop
- classic protocol: resize the short side to 256, centre-crop , normalise with the ImageNet mean (0.485, 0.456, 0.406) and std (0.229, 0.224, 0.225)
Augmentation and transfer learning
The default training augmentation is the Inception-style random-resized crop (a random region covering 8% to 100% of the image, at a random aspect ratio, resized to 224) plus a horizontal flip. Modern recipes add RandAugment, Mixup, CutMix, Label smoothing, Stochastic depth and long schedules. These recipe changes alone account for the 4.7-point gap between the two ResNet-50 checkpoints above (see Data augmentation).
Features learned on ImageNet transfer. To solve a new -class task, load the pretrained backbone and replace the head with a fresh layer. A linear probe freezes the backbone and trains only the head; fine-tuning trains everything with a learning rate well below the pretraining one. Fine-tuning usually wins when there are thousands of labelled images; a linear probe is cheaper and safer with very few.
Calibration
A classifier is calibrated if, among the images it predicts with confidence 0.8, 80% are correct. Guo, Pleiss, Sun and Weinberger ("On Calibration of Modern Neural Networks", ICML 2017) showed that deep, heavily trained networks such as ResNets are over-confident. They measured it with the expected calibration error over confidence bins:
- validation images whose top probability falls in bin (e.g. or 15 equal-width bins)
- fraction correct and mean top probability inside the bin
Their fix is temperature scaling: learn one scalar on the validation set by minimising the negative log-likelihood (NLL), and divide all logits by it. Dividing by a positive constant does not change the order of the logits, so top-1 and top-5 accuracy are unchanged; only the confidences move.
Worked example: one image, five classes
Logits for (tabby cat, tiger cat, lynx, fox, car); the true label is tiger cat.
| Class | z | p at T = 1 | p at T = 2 |
|---|---|---|---|
| tabby cat | 4.0 | 0.7634 | 0.5125 |
| tiger cat (true) | 2.5 | 0.1703 | 0.2421 |
| lynx | 1.0 | 0.0380 | 0.1143 |
| fox | 0.5 | 0.0231 | 0.0891 |
| car | −1.0 | 0.0051 | 0.0421 |
| loss −log p_y | 1.7699 | 1.4185 |
Top-1 is wrong (tabby cat), top-5 is right (tiger cat is rank 2), at both temperatures. keeps the ranking but lowers the loss from 1.770 to 1.419, because the over-confident 0.763 on the wrong class drops to 0.513. Averaged over a validation set, that NLL reduction is what temperature scaling optimises.
1import torch, torch.nn as nn2from torchvision import models, transforms34weights = models.ResNet50_Weights.IMAGENET1K_V25model = models.resnet50(weights=weights)6model.fc = nn.Linear(model.fc.in_features, 10) # 2048 -> 10, new head78train_tf = transforms.Compose([9 transforms.RandomResizedCrop(224), # scale (0.08, 1.0) by default10 transforms.RandomHorizontalFlip(),11 transforms.ToTensor(),12 transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),13])14eval_tf = weights.transforms() # resize, centre-crop 224, normalise1516# backbone at a low rate, fresh head at a higher one17opt = torch.optim.SGD([18 {"params": [p for n, p in model.named_parameters() if not n.startswith("fc.")], "lr": 1e-3},19 {"params": model.fc.parameters(), "lr": 1e-2},20], momentum=0.9, weight_decay=1e-4)21loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)2223@torch.no_grad()24def topk(logits, y, ks=(1, 5)):25 top = logits.topk(max(ks), dim=1).indices # (B, 5)26 hit = top == y[:, None]27 return [hit[:, :k].any(1).float().mean().item() for k in ks]2829def fit_temperature(val_logits, val_y):30 """Temperature scaling (Guo et al., 2017): one scalar T, minimise validation NLL."""31 log_t = torch.zeros(1, requires_grad=True)32 opt_t = torch.optim.LBFGS([log_t], lr=0.1, max_iter=100)33 def closure():34 opt_t.zero_grad()35 loss = nn.functional.cross_entropy(val_logits / log_t.exp(), val_y)36 loss.backward()37 return loss38 opt_t.step(closure)39 return log_t.exp().item()Check yourself
After temperature scaling with T = 2.5, what happens to top-1 and top-5 accuracy?
Check yourself
You have 300 labelled images for a new 5-class task and an ImageNet-pretrained ResNet-50. What is the safest first baseline?
Remember
- ✓Classifier = backbone → global average pooling → linear layer → softmax, trained with cross-entropy; the gradient on the logits is p − onehot(y).
- ✓ImageNet-1k: 1000 classes, 1.28M training and 50k validation images, reported as top-1 and top-5 accuracy.
- ✓Backbones changed (AlexNet, VGG, ResNet, EfficientNet, ViT, ConvNeXt); the head and loss did not.
- ✓Transfer: swap the head for a K-way layer; linear probe (frozen) or fine-tune (all layers, low learning rate).
- ✓Temperature scaling divides logits by one scalar T fitted on validation NLL: ECE drops, accuracy does not change.