Definition
AlexNet is the convolutional network of Krizhevsky, Sutskever and Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" (NeurIPS 2012). Five convolutional layers ( stride 4, then , then three ) with max-pooling, followed by two 4096-unit dense layers and a 1000-way softmax: about 60 million parameters and 650,000 neurons. It was trained on two GPUs, each holding half of the kernels.
Before 2012 the best ImageNet systems were pipelines of hand-designed features (SIFT, Fisher vectors) and linear classifiers. AlexNet won the ILSVRC-2012 classification task with 15.3% top-5 test error; the second-best entry had 26.2%. That gap, more than ten points on a benchmark where yearly gains had been one or two, convinced the field. Within two years nearly every competitive vision system was a CNN.
The architecture itself is a scaled-up LeNet-5: conv and pool stages, then dense layers. What was new was the scale (1.2 million training images, 60M parameters) and a set of practical choices that made that scale trainable: ReLU instead of tanh, dropout in the dense layers, overlapping max-pooling, local response normalisation, heavy data augmentation, and a hand-written CUDA implementation split across two GTX 580 GPUs with 3 GB each. Training took five to six days for about 90 epochs.
Shape, parameter and cost arithmetic
- kernel side, stride and padding. conv1: , ,
- groups: 2 for conv2, conv4 and conv5, whose kernels only see the maps on their own GPU
- multiply-accumulates: the unit torchvision's "GFLOPS" column counts
The paper says the input is , but is not an integer. The maps it reports need a input (or with 2 pixels of padding, which is what torchvision does). The layer table below uses 227.
| layer | output | params | MACs | notes |
|---|---|---|---|---|
| conv1 11×11/4, 96 | 96×55×55 | 34,944 | 105.4M | LRN, then 3×3/2 max-pool → 27×27 |
| conv2 5×5, 256, g=2 | 256×27×27 | 307,456 | 223.9M | LRN, then max-pool → 13×13 |
| conv3 3×3, 384 | 384×13×13 | 885,120 | 149.5M | reads both GPUs' maps |
| conv4 3×3, 384, g=2 | 384×13×13 | 663,936 | 112.1M | |
| conv5 3×3, 256, g=2 | 256×13×13 | 442,624 | 74.8M | max-pool → 6×6 |
| fc6 | 4096 | 37,752,832 | 37.7M | 9216 inputs; dropout 0.5 |
| fc7 | 4096 | 16,781,312 | 16.8M | dropout 0.5 |
| fc8 | 1000 | 4,097,000 | 4.1M | softmax |
The split is lopsided. The five conv layers hold 3.8% of the parameters but do 91.9% of the multiply-adds. The three dense layers hold the other 96.2%, and fc6 alone has 37.8M. So dropout went where the parameters were, and the conv layers are where the GPUs spent their time.
The four ingredients
ReLU. Krizhevsky et al. called units "Rectified Linear Units" and measured that a 4-layer CNN with them reached 25% training error on CIFAR-10 six times faster than the same network with tanh. Non-saturating units were what made a network this size trainable in a week (see ReLU).
Dropout. Each hidden unit of fc6 and fc7 is zeroed with probability 0.5 on every training example, and at test time all units are used with outputs multiplied by 0.5. Without it the network overfit badly. With it, it needed about twice as many iterations to converge (see Dropout).
Overlapping max-pooling. The pooling windows are with stride 2, so neighbouring windows overlap by one row. Compared with non-overlapping windows of stride 2, which give the same output size, this cut top-1 / top-5 error by 0.4% / 0.3%, and the authors observed slightly less overfitting.
Local response normalisation (LRN). After the ReLUs of conv1 and conv2, each activation is divided by a power of the summed squares of the same position in adjacent maps, a form of lateral inhibition. The paper reports it lowered top-1 / top-5 error by 1.4% / 1.2%. Later work, starting with VGGNet, found it does little in deeper networks, and Batch normalization replaced it.
- the ReLU output of kernel at position
- the number of kernels in the layer
- keeps the denominator near when activity is small, so LRN mostly rescales and only damps strong local responses
Worked example: conv1, conv2 and one LRN step
- 1conv1: . Parameters . MACs M.
- 2pool1: . conv2 with padding 2 keeps 27. With each kernel sees maps: parameters. Without the split it would be .
- 3After pool5 the map is values, so fc6 has parameters, 62% of the network.
- 4LRN: five adjacent maps with activations at one position. The sum of squares is , the denominator , and the middle unit becomes .
- 5Scale all five by 10: the sum of squares is , the denominator , and the middle unit becomes instead of . Strong local activity is damped by 16.5%; weak activity is only rescaled.
Data augmentation and the two-GPU split
Each training image is rescaled to and the network sees a random crop, flipped horizontally half the time. The paper counts this as enlarging the training set by a factor of , and without it the network overfit substantially. At test time it averages the softmax over ten patches: the four corner crops and the centre crop, each with its mirror image.
The second augmentation alters colour. PCA is run once on the RGB values of all ImageNet training pixels. Then every time an image is used, one random offset along the principal components is added to all of its pixels:
- eigenvectors and eigenvalues of the RGB covariance matrix
- drawn once per image per epoch; the same shift is added to every pixel
The shift follows the directions in which natural colours actually vary, so it mimics changes in illumination intensity and colour. The paper reports that it reduced top-1 error by over 1%.
The model did not fit on one 3 GB GPU, so each GPU held half of the kernels of every layer: 48 of conv1's 96, 128 of conv2's 256, and so on. The GPUs exchanged activations only at conv3 and at the dense layers. conv2, conv4 and conv5 read only the maps on their own GPU; today this is written groups=2. The authors found the split cut top-1 / top-5 error by 1.7% / 1.2% compared with a net with half as many kernels per conv layer trained on one GPU. They also noticed that the two halves specialised: one GPU's conv1 kernels came out largely colour-agnostic, the other's colour-specific.
At a glance
- Paper
- Krizhevsky, Sutskever, Hinton, NeurIPS 2012
- Size
- 60,965,224 parameters (paper: "60 million"), 0.72 G multiply-adds
- Optimiser
- SGD, batch 128, momentum 0.9, weight decay 0.0005; lr 0.01, divided by 10 three times when validation error plateaued
- Init
- weights ; biases 1 in conv2, conv4, conv5 and the hidden dense layers (so ReLUs start active), 0 elsewhere
- ILSVRC-2010 test
- 37.5% top-1, 17.0% top-5 error
- ILSVRC-2012
- 18.2% top-5 val (1 net); 15.3% test with an ensemble of 7 nets, 2 of them pre-trained on the larger Fall 2011 ImageNet release
- torchvision
alexnet(IMAGENET1K_V1): 61.1M params, 0.71 GFLOPS, 56.522% top-1, 79.066% top-5
1import torch, torch.nn as nn2from torchvision.models import alexnet, AlexNet_Weights34# PyTorch's LocalResponseNorm divides alpha by the window size n,5# so the paper's alpha = 1e-4 is alpha = 5e-4 here.6lrn = lambda: nn.LocalResponseNorm(size=5, alpha=5e-4, beta=0.75, k=2.0)78paper_alexnet = nn.Sequential(9 nn.Conv2d(3, 96, 11, stride=4), nn.ReLU(inplace=True), lrn(), nn.MaxPool2d(3, 2), # 55 -> 2710 nn.Conv2d(96, 256, 5, padding=2, groups=2), nn.ReLU(inplace=True), lrn(), nn.MaxPool2d(3, 2), # 27 -> 1311 nn.Conv2d(256, 384, 3, padding=1), nn.ReLU(inplace=True),12 nn.Conv2d(384, 384, 3, padding=1, groups=2), nn.ReLU(inplace=True),13 nn.Conv2d(384, 256, 3, padding=1, groups=2), nn.ReLU(inplace=True), nn.MaxPool2d(3, 2), # 13 -> 614 nn.Flatten(),15 nn.Dropout(0.5), nn.Linear(256 * 6 * 6, 4096), nn.ReLU(inplace=True),16 nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(inplace=True),17 nn.Linear(4096, 1000),18)19print(paper_alexnet(torch.randn(2, 3, 227, 227)).shape) # torch.Size([2, 1000])20print(sum(p.numel() for p in paper_alexnet.parameters())) # 609652242122tv = alexnet(weights=AlexNet_Weights.IMAGENET1K_V1).eval() # 64-192-384-256-256, no LRN23print(sum(p.numel() for p in tv.parameters())) # 6110084024preprocess = AlexNet_Weights.IMAGENET1K_V1.transforms() # resize 256, centre-crop 224, normaliseUse when
- +Showed that a large CNN trained end to end beats hand-crafted features by a wide margin
- +Simple, sequential layout that is easy to implement and to reason about
- +Cheap by modern standards: 0.72 G multiply-adds per image
- +Its training recipe (ReLU, dropout, augmentation, momentum SGD with weight decay) became the default
Avoid when
- −96% of its 61M parameters are in the dense layers: large to store, prone to overfitting
- −11×11 stride-4 first layer aliases and wastes filters (ZFNet's diagnosis)
- −LRN and the two-GPU split are hardware- and era-specific tricks
- −Far less accurate than later nets of similar cost: torchvision's AlexNet reaches 56.5% top-1
Check yourself
conv2 of AlexNet maps 96 input maps to 256 output maps with 5×5 kernels, split across two GPUs (groups = 2). How many parameters does it have?
Check yourself
Where would dropout do the most good in AlexNet, and why did the authors put it there?
Remember
- ✓5 conv + 3 dense layers, ≈ 61M parameters, 0.72 G multiply-adds; 227×227 input (the paper's "224" needs padding).
- ✓ILSVRC-2012: 15.3% top-5 test error vs 26.2% for the runner-up, the result that turned vision to deep learning.
- ✓ReLU (6× faster to train than tanh), dropout 0.5 in fc6/fc7, overlapping 3×3/2 max-pooling, LRN after conv1 and conv2.
- ✓Augmentation: random 224 crops of 256 images plus flips (×2048), and PCA colour shifts with α ~ N(0, 0.1²).
- ✓The two-GPU split is groups = 2 in conv2/4/5: it halves those layers' parameters and removes 36% of the compute of the same widths ungrouped.
- ✓The dense layers hold 96% of the parameters; the conv layers do 92% of the compute.