# Image classification

> Map an image to one of K labels: a backbone, global pooling, a linear layer and a softmax trained with cross-entropy. ImageNet-1k (1.28M train / 50k val, 1000 classes) is the benchmark, scored by top-1 and top-5 accuracy.

Source: https://www.mlfoundry.in/learn/image-classification  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 7 min read · updated 2026-10-15

**Image classification:** is the task of assigning a whole image one label from a fixed set of $K$ classes. The model outputs a probability distribution $p\in\Delta^{K-1}$, and the prediction is $\arg\max_k p_k$. The reference benchmark is the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 subset of ImageNet (Deng et al., CVPR 2009), now usually called **ImageNet-1k**.

Input: an RGB tensor, usually $3\times224\times224$ after resizing and cropping. Output: $K$ scores. The task is hard because one label covers huge variation in pose, scale, lighting, occlusion and background, and some classes differ in fine detail, such as ImageNet's many dog breeds. Before 2012 the best systems used hand-built features (SIFT, HOG, Fisher vectors) and a linear SVM. **AlexNet** cut ILSVRC-2012 top-5 error from 26.2% to 15.3% and ended that era.

## The standard recipe

Every modern classifier has the same layout. A **backbone** turns the image into a feature map, global average pooling (GAP) turns the map into one vector $h$, and a linear layer gives logits $z=Wh+b$. Softmax turns the logits into probabilities. Training minimises the cross-entropy of the true label $y$:

$$
p_k=\frac{e^{z_k/T}}{\sum_{j=1}^{K}e^{z_j/T}},\qquad \mathcal L=-\log p_y,\qquad \frac{\partial\mathcal L}{\partial z_k}=p_k-\mathbb 1[k=y]\quad(T=1)
$$

- $z\in\mathbb R^K$: logits from the linear head, $K=1000$ for ImageNet-1k
- $T$: temperature; $T=1$ in training, tuned after training for calibration
- $y$: index of the true class

> Figure: A ResNet-50 classifier. Four stages shrink the 224 × 224 input to a 7 × 7 × 2048 map (stride 32), global average pooling gives a 2048-vector, and a 2048 × 1000 linear layer gives the logits. For transfer learning only the head is swapped; the scores shown are illustrative.

The backbone is where architectures differ. The arc runs from **LeNet-5** (digits, 1998) to **AlexNet** (2012), **VGGNet** and **GoogLeNet / Inception** (2014), **ResNet** (2015, residual blocks that make 50 to 152 layers trainable), **EfficientNet** (compound scaling) and **ConvNeXt** among CNNs. **Vision Transformers (ViT)** split the image into $16\times16$ patches and replace convolution with self-attention; **Swin Transformer** brings back locality and a feature pyramid. The head and the loss stay the same throughout.

**Benchmark and reference numbers**

- **ImageNet-1k:** 1000 classes; 1,281,167 training images, 50,000 validation (50 per class), 100,000 test; results are normally reported on the validation set
- **Top-1 / top-5:** top-1: the arg-max is the true label; top-5: the true label is among the 5 highest scores. Top-5 was the ILSVRC metric because many images contain several objects
- **ILSVRC winners (top-5 error):** AlexNet 15.3% (2012); ResNet ensemble 3.57% (2015)
- **ResNet-50:** 25.6M parameters, 4.1G multiply-adds; torchvision weights: 76.13% top-1 (V1) and 80.86% (V2, same architecture, newer training recipe)
- **Evaluation crop:** classic protocol: resize the short side to 256, centre-crop $224\times224$, normalise with the ImageNet mean (0.485, 0.456, 0.406) and std (0.229, 0.224, 0.225)

## Augmentation and transfer learning

The default training augmentation is the Inception-style random-resized crop (a random region covering 8% to 100% of the image, at a random aspect ratio, resized to 224) plus a horizontal flip. Modern recipes add RandAugment, **Mixup**, **CutMix**, **Label smoothing**, **Stochastic depth** and long schedules. These recipe changes alone account for the 4.7-point gap between the two ResNet-50 checkpoints above (see **Data augmentation**).

Features learned on ImageNet transfer. To solve a new $K$-class task, load the pretrained backbone and replace the $2048\times1000$ head with a fresh $2048\times K$ layer. A **linear probe** freezes the backbone and trains only the head; **fine-tuning** trains everything with a learning rate well below the pretraining one. Fine-tuning usually wins when there are thousands of labelled images; a linear probe is cheaper and safer with very few.

## Calibration

A classifier is **calibrated** if, among the images it predicts with confidence 0.8, 80% are correct. Guo, Pleiss, Sun and Weinberger ("On Calibration of Modern Neural Networks", ICML 2017) showed that deep, heavily trained networks such as ResNets are over-confident. They measured it with the expected calibration error over $M$ confidence bins:

$$
\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\,\big|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\big|
$$

- $B_m$: validation images whose top probability $\max_k p_k$ falls in bin $m$ (e.g. $M=10$ or 15 equal-width bins)
- $\operatorname{acc},\ \operatorname{conf}$: fraction correct and mean top probability inside the bin

Their fix is **temperature scaling**: learn one scalar $T>0$ on the validation set by minimising the negative log-likelihood (NLL), and divide all logits by it. Dividing by a positive constant does not change the order of the logits, so top-1 and top-5 accuracy are unchanged; only the confidences move.

> Figure: A synthetic 600-image, 10-class validation set whose logits are 2.5× too large. At T = 1, top-1 is 73.7%, top-5 94.7% and mean confidence 96.4%: NLL 1.662, ECE 0.227, and the reliability bars sit far below the diagonal. "Fit T" finds T = 2.55: confidence falls to 74.8%, NLL to 0.964 and ECE to 0.034, while top-1 and top-5 do not move. At T = 4 the model becomes under-confident (confidence 53.2%, ECE 0.205). The "cat → dog" image is 99.3% dog at T = 1 and 70.7% at T = 2.55; cat stays rank 3, a top-5 hit.

## Worked example: one image, five classes

Logits $z=(4.0,\ 2.5,\ 1.0,\ 0.5,\ -1.0)$ for (tabby cat, tiger cat, lynx, fox, car); the true label is **tiger cat**.

Softmax at T = 1 and T = 2 (sums of exponentials 71.5155 and 14.4187)

| Class | z | p at T = 1 | p at T = 2 |
| --- | --- | --- | --- |
| tabby cat | 4.0 | 0.7634 | 0.5125 |
| tiger cat (true) | 2.5 | 0.1703 | 0.2421 |
| lynx | 1.0 | 0.0380 | 0.1143 |
| fox | 0.5 | 0.0231 | 0.0891 |
| car | −1.0 | 0.0051 | 0.0421 |
| loss −log p_y |  | 1.7699 | 1.4185 |

Top-1 is wrong (tabby cat), top-5 is right (tiger cat is rank 2), at both temperatures. $T=2$ keeps the ranking but lowers the loss from 1.770 to 1.419, because the over-confident 0.763 on the wrong class drops to 0.513. Averaged over a validation set, that NLL reduction is what temperature scaling optimises.

```python
import torch, torch.nn as nn
from torchvision import models, transforms

weights = models.ResNet50_Weights.IMAGENET1K_V2
model = models.resnet50(weights=weights)
model.fc = nn.Linear(model.fc.in_features, 10)           # 2048 -> 10, new head

train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224),                    # scale (0.08, 1.0) by default
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
eval_tf = weights.transforms()                            # resize, centre-crop 224, normalise

# backbone at a low rate, fresh head at a higher one
opt = torch.optim.SGD([
    {"params": [p for n, p in model.named_parameters() if not n.startswith("fc.")], "lr": 1e-3},
    {"params": model.fc.parameters(), "lr": 1e-2},
], momentum=0.9, weight_decay=1e-4)
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)

@torch.no_grad()
def topk(logits, y, ks=(1, 5)):
    top = logits.topk(max(ks), dim=1).indices             # (B, 5)
    hit = top == y[:, None]
    return [hit[:, :k].any(1).float().mean().item() for k in ks]

def fit_temperature(val_logits, val_y):
    """Temperature scaling (Guo et al., 2017): one scalar T, minimise validation NLL."""
    log_t = torch.zeros(1, requires_grad=True)
    opt_t = torch.optim.LBFGS([log_t], lr=0.1, max_iter=100)
    def closure():
        opt_t.zero_grad()
        loss = nn.functional.cross_entropy(val_logits / log_t.exp(), val_y)
        loss.backward()
        return loss
    opt_t.step(closure)
    return log_t.exp().item()
```

Fine-tuning a pretrained ResNet-50 on a new 10-class task with torchvision (not run here: needs torch, torchvision and a dataset)

> **Leakage between pretraining and evaluation:** Fine-tuning results are only comparable when the test images were never seen in pretraining, and when the validation set used for early stopping or temperature fitting is not the test set. Fitting $T$ on test data and reporting the ECE there inflates calibration as surely as tuning hyperparameters on test inflates accuracy.

> **Accuracy and calibration are separate axes:** A monotone rescaling of the logits cannot change which class wins, so accuracy and calibration can be fixed independently. Training tricks that soften targets (**Label smoothing**, **Mixup**) also change calibration, so measure ECE after them rather than assuming.

**Question:** After temperature scaling with T = 2.5, what happens to top-1 and top-5 accuracy?

- A. Both rise
- B. Both fall
- C. Neither changes
- D. Top-1 is unchanged, top-5 rises

**Answer:** C. Neither changes — $z_k/T$ with $T>0$ preserves the order of the logits, so the arg-max and the five highest classes are the same. Only the probabilities, and therefore NLL and ECE, change.

**Question:** You have 300 labelled images for a new 5-class task and an ImageNet-pretrained ResNet-50. What is the safest first baseline?

- A. Train ResNet-50 from scratch
- B. A linear probe: freeze the backbone, train a new 2048 × 5 head
- C. Keep the 1000-way head and map classes by name
- D. Fine-tune all layers at the ImageNet learning rate

**Answer:** B. A linear probe: freeze the backbone, train a new 2048 × 5 head — With very little data a frozen backbone plus a new linear head is cheap and hard to overfit. Full fine-tuning at a small learning rate often adds accuracy once there is more data; training from scratch or at the pretraining rate wastes the pretrained features.

**Key points**

- Classifier = backbone → global average pooling → linear layer → softmax, trained with cross-entropy; the gradient on the logits is p − onehot(y).
- ImageNet-1k: 1000 classes, 1.28M training and 50k validation images, reported as top-1 and top-5 accuracy.
- Backbones changed (AlexNet, VGG, ResNet, EfficientNet, ViT, ConvNeXt); the head and loss did not.
- Transfer: swap the head for a K-way layer; linear probe (frozen) or fine-tune (all layers, low learning rate).
- Temperature scaling divides logits by one scalar T fitted on validation NLL: ECE drops, accuracy does not change.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
