AlexNet

The 2012 ImageNet winner: 5 conv + 3 dense layers, about 60M parameters, trained on two GPUs. ReLU, dropout, overlapping pooling and heavy augmentation cut ILSVRC-2012 top-5 error to 15.3% from 26.2%, and started the deep learning era in vision.

13 min read1141 wordscore0 viewsupdated 2026-10-11

Definition

AlexNet is the convolutional network of Krizhevsky, Sutskever and Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" (NeurIPS 2012). Five convolutional layers (11×1111\times11 stride 4, then 5×55\times5, then three 3×33\times3) with max-pooling, followed by two 4096-unit dense layers and a 1000-way softmax: about 60 million parameters and 650,000 neurons. It was trained on two GPUs, each holding half of the kernels.

Before 2012 the best ImageNet systems were pipelines of hand-designed features (SIFT, Fisher vectors) and linear classifiers. AlexNet won the ILSVRC-2012 classification task with 15.3% top-5 test error; the second-best entry had 26.2%. That gap, more than ten points on a benchmark where yearly gains had been one or two, convinced the field. Within two years nearly every competitive vision system was a CNN.

The architecture itself is a scaled-up LeNet-5: conv and pool stages, then dense layers. What was new was the scale (1.2 million training images, 60M parameters) and a set of practical choices that made that scale trainable: ReLU instead of tanh, dropout in the dense layers, overlapping max-pooling, local response normalisation, heavy data augmentation, and a hand-written CUDA implementation split across two GTX 580 GPUs with 3 GB each. Training took five to six days for about 90 epochs.

Shape, parameter and cost arithmetic

nout=⌊nin+2p−ks⌋+1,P=Cing k2 Cout+Cout,MACs=nout2⋅Cing k2 Coutn_{out}=\Big\lfloor\frac{n_{in}+2p-k}{s}\Big\rfloor+1,\qquad P=\frac{C_{in}}{g}\,k^2\,C_{out}+C_{out},\qquad \text{MACs}=n_{out}^2\cdot\frac{C_{in}}{g}\,k^2\,C_{out}
k, s, pk,\,s,\,p
kernel side, stride and padding. conv1: k=11k=11, s=4s=4, p=0p=0
gg
groups: 2 for conv2, conv4 and conv5, whose kernels only see the maps on their own GPU
MACs\text{MACs}
multiply-accumulates: the unit torchvision's "GFLOPS" column counts

The paper says the input is 224×224224\times224, but (224−11)/4+1(224-11)/4+1 is not an integer. The 55×5555\times55 maps it reports need a 227×227227\times227 input (or 224224 with 2 pixels of padding, which is what torchvision does). The layer table below uses 227.

layeroutputparamsMACsnotes
conv1 11×11/4, 9696×55×5534,944105.4MLRN, then 3×3/2 max-pool → 27×27
conv2 5×5, 256, g=2256×27×27307,456223.9MLRN, then max-pool → 13×13
conv3 3×3, 384384×13×13885,120149.5Mreads both GPUs' maps
conv4 3×3, 384, g=2384×13×13663,936112.1M
conv5 3×3, 256, g=2256×13×13442,62474.8Mmax-pool → 6×6
fc6409637,752,83237.7M9216 inputs; dropout 0.5
fc7409616,781,31216.8Mdropout 0.5
fc810004,097,0004.1Msoftmax
AlexNet as in the paper (2 GPUs), computed layer by layer. Totals: 60,965,224 parameters and 0.72 G multiply-adds per image

The split is lopsided. The five conv layers hold 3.8% of the parameters but do 91.9% of the multiply-adds. The three dense layers hold the other 96.2%, and fc6 alone has 37.8M. So dropout went where the parameters were, and the conv layers are where the GPUs spent their time.

AlexNet · paper, 2 GPUs · click a layer60,965,224 params · 0.72 GMACs227²·3inconv155²96lrn155²96pool127²96conv227²256lrn227²256pool213²256conv313²384conv413²384conv513²256pool56²256fc614096fc714096fc811000layerconv 5×5, 256 maps, s1 p2 g2in → out96×27×27 → 256×27×27params307,456 (0.5%)MACs223.95M (30.9%)recept. field51×51 px, stride 8each GPU's 128 kernels see only its own 48 maps2-GPU split saves 1,413,120 params,411M MACs (36.2% of 1-GPU compute)32² image (≙256²) → 28² crop (≙224²)offset (2,2) · paper: 2,048 viewsRGB PCA: λ = 0.1330, 0.0391, 0.0037p₁ = (0.26, 0.22, 0.94)δ = Σ αᵢλᵢpᵢ = (+5.2, +4.4, +19.2)/255conv layers: 3.8% of params, 91.9% of compute · dense: 96.2% of paramsparamsMACs■ conv ■ LRN ■ max-pool ■ dense · block height ∝ log(map size)
fig.AlexNet layer by layer. Click a block for its output shape, parameters, multiply-adds and receptive field (conv5 units see 163×163 pixels). The variant buttons recompute everything. The paper's two-GPU network has 60,965,224 parameters and 0.72 G multiply-adds. The same widths on one GPU, with conv2/4/5 reading all maps, have 1,413,120 more parameters and 411M more multiply-adds: the split removed 36.2% of the compute. torchvision's single-GPU AlexNet (widths 64-192-384-256-256) has 61,100,840 parameters. On the right, the training augmentation on a small generated image: drag the crop window, flip it, and set the PCA colour weights α. The figure does PCA on this image's own pixels (λ = 0.133, 0.039, 0.004), so α₁ = 0.6 shifts every pixel by (+5.2, +4.4, +19.2)/255.

The four ingredients

ReLU. Krizhevsky et al. called max⁡(0,x)\max(0,x) units "Rectified Linear Units" and measured that a 4-layer CNN with them reached 25% training error on CIFAR-10 six times faster than the same network with tanh. Non-saturating units were what made a network this size trainable in a week (see ReLU).

Dropout. Each hidden unit of fc6 and fc7 is zeroed with probability 0.5 on every training example, and at test time all units are used with outputs multiplied by 0.5. Without it the network overfit badly. With it, it needed about twice as many iterations to converge (see Dropout).

Overlapping max-pooling. The pooling windows are 3×33\times3 with stride 2, so neighbouring windows overlap by one row. Compared with non-overlapping 2×22\times2 windows of stride 2, which give the same output size, this cut top-1 / top-5 error by 0.4% / 0.3%, and the authors observed slightly less overfitting.

Local response normalisation (LRN). After the ReLUs of conv1 and conv2, each activation is divided by a power of the summed squares of the same position in n=5n=5 adjacent maps, a form of lateral inhibition. The paper reports it lowered top-1 / top-5 error by 1.4% / 1.2%. Later work, starting with VGGNet, found it does little in deeper networks, and Batch normalization replaced it.

bx,yi=ax,yi/(k+α ⁣ ⁣∑j=max⁡(0,i−n/2)min⁡(N−1, i+n/2) ⁣ ⁣(ax,yj)2)β,k=2,  n=5,  α=10−4,  β=0.75b^i_{x,y}=a^i_{x,y}\Big/\Big(k+\alpha\!\!\sum_{j=\max(0,i-n/2)}^{\min(N-1,\,i+n/2)}\!\!\big(a^j_{x,y}\big)^2\Big)^{\beta},\qquad k=2,\;n=5,\;\alpha=10^{-4},\;\beta=0.75
ax,yia^i_{x,y}
the ReLU output of kernel ii at position (x,y)(x,y)
NN
the number of kernels in the layer
k=2k=2
keeps the denominator near 20.75=1.682^{0.75}=1.68 when activity is small, so LRN mostly rescales and only damps strong local responses

Worked example: conv1, conv2 and one LRN step

  1. 1conv1: ⌊(227−11)/4⌋+1=55\lfloor(227-11)/4\rfloor+1=55. Parameters 11⋅11⋅3⋅96+96=34,94411\cdot11\cdot3\cdot96+96=34{,}944. MACs 552⋅363⋅96=105.455^2\cdot363\cdot96=105.4M.
  2. 2pool1: ⌊(55−3)/2⌋+1=27\lfloor(55-3)/2\rfloor+1=27. conv2 with padding 2 keeps 27. With g=2g=2 each kernel sees 96/2=4896/2=48 maps: 5⋅5⋅48⋅256+256=307,4565\cdot5\cdot48\cdot256+256=307{,}456 parameters. Without the split it would be 614,656614{,}656.
  3. 3After pool5 the map is 256×6×6=9,216256\times6\times6=9{,}216 values, so fc6 has 9216⋅4096+4096=37,752,8329216\cdot4096+4096=37{,}752{,}832 parameters, 62% of the network.
  4. 4LRN: five adjacent maps with activations (1,2,3,4,5)(1,2,3,4,5) at one position. The sum of squares is 5555, the denominator (2+10−4⋅55)0.75=1.6853(2+10^{-4}\cdot55)^{0.75}=1.6853, and the middle unit becomes 3/1.6853=1.7803/1.6853=1.780.
  5. 5Scale all five by 10: the sum of squares is 5,5005{,}500, the denominator 2.01792.0179, and the middle unit becomes 30/2.0179=14.8730/2.0179=14.87 instead of 17.8017.80. Strong local activity is damped by 16.5%; weak activity is only rescaled.

Data augmentation and the two-GPU split

Each training image is rescaled to 256×256256\times256 and the network sees a random 224×224224\times224 crop, flipped horizontally half the time. The paper counts this as enlarging the training set by a factor of 32⋅32⋅2=204832\cdot32\cdot2=2048, and without it the network overfit substantially. At test time it averages the softmax over ten patches: the four corner crops and the centre crop, each with its mirror image.

The second augmentation alters colour. PCA is run once on the RGB values of all ImageNet training pixels. Then every time an image is used, one random offset along the principal components is added to all of its pixels:

Ixy′=Ixy+[p1,p2,p3][α1λ1,  α2λ2,  α3λ3]⊤,αi∼N(0, 0.12)I'_{xy}=I_{xy}+\big[\mathbf p_1,\mathbf p_2,\mathbf p_3\big]\big[\alpha_1\lambda_1,\;\alpha_2\lambda_2,\;\alpha_3\lambda_3\big]^\top,\qquad \alpha_i\sim\mathcal N(0,\,0.1^2)
pi, λi\mathbf p_i,\,\lambda_i
eigenvectors and eigenvalues of the 3×33\times3 RGB covariance matrix
αi\alpha_i
drawn once per image per epoch; the same shift is added to every pixel

The shift follows the directions in which natural colours actually vary, so it mimics changes in illumination intensity and colour. The paper reports that it reduced top-1 error by over 1%.

The model did not fit on one 3 GB GPU, so each GPU held half of the kernels of every layer: 48 of conv1's 96, 128 of conv2's 256, and so on. The GPUs exchanged activations only at conv3 and at the dense layers. conv2, conv4 and conv5 read only the maps on their own GPU; today this is written groups=2. The authors found the split cut top-1 / top-5 error by 1.7% / 1.2% compared with a net with half as many kernels per conv layer trained on one GPU. They also noticed that the two halves specialised: one GPU's conv1 kernels came out largely colour-agnostic, the other's colour-specific.

At a glance

Paper
Krizhevsky, Sutskever, Hinton, NeurIPS 2012
Size
60,965,224 parameters (paper: "60 million"), 0.72 G multiply-adds
Optimiser
SGD, batch 128, momentum 0.9, weight decay 0.0005; lr 0.01, divided by 10 three times when validation error plateaued
Init
weights N(0,0.012)\mathcal N(0,0.01^2); biases 1 in conv2, conv4, conv5 and the hidden dense layers (so ReLUs start active), 0 elsewhere
ILSVRC-2010 test
37.5% top-1, 17.0% top-5 error
ILSVRC-2012
18.2% top-5 val (1 net); 15.3% test with an ensemble of 7 nets, 2 of them pre-trained on the larger Fall 2011 ImageNet release
torchvision
alexnet (IMAGENET1K_V1): 61.1M params, 0.71 GFLOPS, 56.522% top-1, 79.066% top-5
python
1import torch, torch.nn as nn
2from torchvision.models import alexnet, AlexNet_Weights
3
4# PyTorch's LocalResponseNorm divides alpha by the window size n,
5# so the paper's alpha = 1e-4 is alpha = 5e-4 here.
6lrn = lambda: nn.LocalResponseNorm(size=5, alpha=5e-4, beta=0.75, k=2.0)
7
8paper_alexnet = nn.Sequential(
9 nn.Conv2d(3, 96, 11, stride=4), nn.ReLU(inplace=True), lrn(), nn.MaxPool2d(3, 2), # 55 -> 27
10 nn.Conv2d(96, 256, 5, padding=2, groups=2), nn.ReLU(inplace=True), lrn(), nn.MaxPool2d(3, 2), # 27 -> 13
11 nn.Conv2d(256, 384, 3, padding=1), nn.ReLU(inplace=True),
12 nn.Conv2d(384, 384, 3, padding=1, groups=2), nn.ReLU(inplace=True),
13 nn.Conv2d(384, 256, 3, padding=1, groups=2), nn.ReLU(inplace=True), nn.MaxPool2d(3, 2), # 13 -> 6
14 nn.Flatten(),
15 nn.Dropout(0.5), nn.Linear(256 * 6 * 6, 4096), nn.ReLU(inplace=True),
16 nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(inplace=True),
17 nn.Linear(4096, 1000),
18)
19print(paper_alexnet(torch.randn(2, 3, 227, 227)).shape) # torch.Size([2, 1000])
20print(sum(p.numel() for p in paper_alexnet.parameters())) # 60965224
21
22tv = alexnet(weights=AlexNet_Weights.IMAGENET1K_V1).eval() # 64-192-384-256-256, no LRN
23print(sum(p.numel() for p in tv.parameters())) # 61100840
24preprocess = AlexNet_Weights.IMAGENET1K_V1.transforms() # resize 256, centre-crop 224, normalise
The paper's two-GPU AlexNet as one module (groups=2), and torchvision's single-GPU variant

Use when

  • +Showed that a large CNN trained end to end beats hand-crafted features by a wide margin
  • +Simple, sequential layout that is easy to implement and to reason about
  • +Cheap by modern standards: 0.72 G multiply-adds per image
  • +Its training recipe (ReLU, dropout, augmentation, momentum SGD with weight decay) became the default

Avoid when

  • −96% of its 61M parameters are in the dense layers: large to store, prone to overfitting
  • −11×11 stride-4 first layer aliases and wastes filters (ZFNet's diagnosis)
  • −LRN and the two-GPU split are hardware- and era-specific tricks
  • −Far less accurate than later nets of similar cost: torchvision's AlexNet reaches 56.5% top-1

Check yourself

conv2 of AlexNet maps 96 input maps to 256 output maps with 5×5 kernels, split across two GPUs (groups = 2). How many parameters does it have?

Check yourself

Where would dropout do the most good in AlexNet, and why did the authors put it there?

Remember

  • ✓5 conv + 3 dense layers, ≈ 61M parameters, 0.72 G multiply-adds; 227×227 input (the paper's "224" needs padding).
  • ✓ILSVRC-2012: 15.3% top-5 test error vs 26.2% for the runner-up, the result that turned vision to deep learning.
  • ✓ReLU (6× faster to train than tanh), dropout 0.5 in fc6/fc7, overlapping 3×3/2 max-pooling, LRN after conv1 and conv2.
  • ✓Augmentation: random 224 crops of 256 images plus flips (×2048), and PCA colour shifts with α ~ N(0, 0.1²).
  • ✓The two-GPU split is groups = 2 in conv2/4/5: it halves those layers' parameters and removes 36% of the compute of the same widths ungrouped.
  • ✓The dense layers hold 96% of the parameters; the conv layers do 92% of the compute.
← back to Deep Learning roadmap