# AlexNet

> The 2012 ImageNet winner: 5 conv + 3 dense layers, about 60M parameters, trained on two GPUs. ReLU, dropout, overlapping pooling and heavy augmentation cut ILSVRC-2012 top-5 error to 15.3% from 26.2%, and started the deep learning era in vision.

Source: https://www.mlfoundry.in/learn/alexnet  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 13 min read · updated 2026-10-11

**AlexNet:** is the convolutional network of Krizhevsky, Sutskever and Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" (NeurIPS 2012). Five convolutional layers ($11\times11$ stride 4, then $5\times5$, then three $3\times3$) with max-pooling, followed by two 4096-unit dense layers and a 1000-way softmax: about 60 million parameters and 650,000 neurons. It was trained on two GPUs, each holding half of the kernels.

Before 2012 the best ImageNet systems were pipelines of hand-designed features (SIFT, Fisher vectors) and linear classifiers. AlexNet won the ILSVRC-2012 classification task with **15.3% top-5 test error**; the second-best entry had **26.2%**. That gap, more than ten points on a benchmark where yearly gains had been one or two, convinced the field. Within two years nearly every competitive vision system was a CNN.

The architecture itself is a scaled-up **LeNet-5**: conv and pool stages, then dense layers. What was new was the scale (1.2 million training images, 60M parameters) and a set of practical choices that made that scale trainable: ReLU instead of tanh, dropout in the dense layers, overlapping max-pooling, local response normalisation, heavy data augmentation, and a hand-written CUDA implementation split across two GTX 580 GPUs with 3 GB each. Training took five to six days for about 90 epochs.

## Shape, parameter and cost arithmetic

$$
n_{out}=\Big\lfloor\frac{n_{in}+2p-k}{s}\Big\rfloor+1,\qquad P=\frac{C_{in}}{g}\,k^2\,C_{out}+C_{out},\qquad \text{MACs}=n_{out}^2\cdot\frac{C_{in}}{g}\,k^2\,C_{out}
$$

- $k,\,s,\,p$: kernel side, stride and padding. conv1: $k=11$, $s=4$, $p=0$
- $g$: groups: 2 for conv2, conv4 and conv5, whose kernels only see the maps on their own GPU
- $\text{MACs}$: multiply-accumulates: the unit torchvision's "GFLOPS" column counts

The paper says the input is $224\times224$, but $(224-11)/4+1$ is not an integer. The $55\times55$ maps it reports need a $227\times227$ input (or $224$ with 2 pixels of padding, which is what torchvision does). The layer table below uses 227.

AlexNet as in the paper (2 GPUs), computed layer by layer. Totals: 60,965,224 parameters and 0.72 G multiply-adds per image

| layer | output | params | MACs | notes |
| --- | --- | --- | --- | --- |
| conv1 11×11/4, 96 | 96×55×55 | 34,944 | 105.4M | LRN, then 3×3/2 max-pool → 27×27 |
| conv2 5×5, 256, g=2 | 256×27×27 | 307,456 | 223.9M | LRN, then max-pool → 13×13 |
| conv3 3×3, 384 | 384×13×13 | 885,120 | 149.5M | reads both GPUs' maps |
| conv4 3×3, 384, g=2 | 384×13×13 | 663,936 | 112.1M |  |
| conv5 3×3, 256, g=2 | 256×13×13 | 442,624 | 74.8M | max-pool → 6×6 |
| fc6 | 4096 | 37,752,832 | 37.7M | 9216 inputs; dropout 0.5 |
| fc7 | 4096 | 16,781,312 | 16.8M | dropout 0.5 |
| fc8 | 1000 | 4,097,000 | 4.1M | softmax |

The split is lopsided. The five conv layers hold **3.8%** of the parameters but do **91.9%** of the multiply-adds. The three dense layers hold the other 96.2%, and fc6 alone has 37.8M. So dropout went where the parameters were, and the conv layers are where the GPUs spent their time.

> Figure: AlexNet layer by layer. Click a block for its output shape, parameters, multiply-adds and receptive field (conv5 units see 163×163 pixels). The variant buttons recompute everything. The paper's two-GPU network has 60,965,224 parameters and 0.72 G multiply-adds. The same widths on one GPU, with conv2/4/5 reading all maps, have 1,413,120 more parameters and 411M more multiply-adds: the split removed 36.2% of the compute. torchvision's single-GPU AlexNet (widths 64-192-384-256-256) has 61,100,840 parameters. On the right, the training augmentation on a small generated image: drag the crop window, flip it, and set the PCA colour weights α. The figure does PCA on this image's own pixels (λ = 0.133, 0.039, 0.004), so α₁ = 0.6 shifts every pixel by (+5.2, +4.4, +19.2)/255.

## The four ingredients

**ReLU.** Krizhevsky et al. called $\max(0,x)$ units "Rectified Linear Units" and measured that a 4-layer CNN with them reached 25% training error on CIFAR-10 **six times faster** than the same network with tanh. Non-saturating units were what made a network this size trainable in a week (see **ReLU**).

**Dropout.** Each hidden unit of fc6 and fc7 is zeroed with probability 0.5 on every training example, and at test time all units are used with outputs multiplied by 0.5. Without it the network overfit badly. With it, it needed about twice as many iterations to converge (see **Dropout**).

**Overlapping max-pooling.** The pooling windows are $3\times3$ with stride 2, so neighbouring windows overlap by one row. Compared with non-overlapping $2\times2$ windows of stride 2, which give the same output size, this cut top-1 / top-5 error by 0.4% / 0.3%, and the authors observed slightly less overfitting.

**Local response normalisation (LRN).** After the ReLUs of conv1 and conv2, each activation is divided by a power of the summed squares of the same position in $n=5$ adjacent maps, a form of lateral inhibition. The paper reports it lowered top-1 / top-5 error by 1.4% / 1.2%. Later work, starting with **VGGNet**, found it does little in deeper networks, and **Batch normalization** replaced it.

$$
b^i_{x,y}=a^i_{x,y}\Big/\Big(k+\alpha\!\!\sum_{j=\max(0,i-n/2)}^{\min(N-1,\,i+n/2)}\!\!\big(a^j_{x,y}\big)^2\Big)^{\beta},\qquad k=2,\;n=5,\;\alpha=10^{-4},\;\beta=0.75
$$

- $a^i_{x,y}$: the ReLU output of kernel $i$ at position $(x,y)$
- $N$: the number of kernels in the layer
- $k=2$: keeps the denominator near $2^{0.75}=1.68$ when activity is small, so LRN mostly rescales and only damps strong local responses

**Worked example: conv1, conv2 and one LRN step**

1. conv1: $\lfloor(227-11)/4\rfloor+1=55$. Parameters $11\cdot11\cdot3\cdot96+96=34{,}944$. MACs $55^2\cdot363\cdot96=105.4$M.
2. pool1: $\lfloor(55-3)/2\rfloor+1=27$. conv2 with padding 2 keeps 27. With $g=2$ each kernel sees $96/2=48$ maps: $5\cdot5\cdot48\cdot256+256=307{,}456$ parameters. Without the split it would be $614{,}656$.
3. After pool5 the map is $256\times6\times6=9{,}216$ values, so fc6 has $9216\cdot4096+4096=37{,}752{,}832$ parameters, 62% of the network.
4. LRN: five adjacent maps with activations $(1,2,3,4,5)$ at one position. The sum of squares is $55$, the denominator $(2+10^{-4}\cdot55)^{0.75}=1.6853$, and the middle unit becomes $3/1.6853=1.780$.
5. Scale all five by 10: the sum of squares is $5{,}500$, the denominator $2.0179$, and the middle unit becomes $30/2.0179=14.87$ instead of $17.80$. Strong local activity is damped by 16.5%; weak activity is only rescaled.

## Data augmentation and the two-GPU split

Each training image is rescaled to $256\times256$ and the network sees a random $224\times224$ crop, flipped horizontally half the time. The paper counts this as enlarging the training set by a factor of $32\cdot32\cdot2=2048$, and without it the network overfit substantially. At test time it averages the softmax over ten patches: the four corner crops and the centre crop, each with its mirror image.

The second augmentation alters colour. PCA is run once on the RGB values of all ImageNet training pixels. Then every time an image is used, one random offset along the principal components is added to all of its pixels:

$$
I'_{xy}=I_{xy}+\big[\mathbf p_1,\mathbf p_2,\mathbf p_3\big]\big[\alpha_1\lambda_1,\;\alpha_2\lambda_2,\;\alpha_3\lambda_3\big]^\top,\qquad \alpha_i\sim\mathcal N(0,\,0.1^2)
$$

- $\mathbf p_i,\,\lambda_i$: eigenvectors and eigenvalues of the $3\times3$ RGB covariance matrix
- $\alpha_i$: drawn once per image per epoch; the same shift is added to every pixel

The shift follows the directions in which natural colours actually vary, so it mimics changes in illumination intensity and colour. The paper reports that it reduced top-1 error by over 1%.

The model did not fit on one 3 GB GPU, so each GPU held half of the kernels of every layer: 48 of conv1's 96, 128 of conv2's 256, and so on. The GPUs exchanged activations only at conv3 and at the dense layers. conv2, conv4 and conv5 read only the maps on their own GPU; today this is written `groups=2`. The authors found the split cut top-1 / top-5 error by 1.7% / 1.2% compared with a net with half as many kernels per conv layer trained on one GPU. They also noticed that the two halves specialised: one GPU's conv1 kernels came out largely colour-agnostic, the other's colour-specific.

**At a glance**

- **Paper:** Krizhevsky, Sutskever, Hinton, NeurIPS 2012
- **Size:** 60,965,224 parameters (paper: "60 million"), 0.72 G multiply-adds
- **Optimiser:** SGD, batch 128, momentum 0.9, weight decay 0.0005; lr 0.01, divided by 10 three times when validation error plateaued
- **Init:** weights $\mathcal N(0,0.01^2)$; biases 1 in conv2, conv4, conv5 and the hidden dense layers (so ReLUs start active), 0 elsewhere
- **ILSVRC-2010 test:** 37.5% top-1, 17.0% top-5 error
- **ILSVRC-2012:** 18.2% top-5 val (1 net); 15.3% test with an ensemble of 7 nets, 2 of them pre-trained on the larger Fall 2011 ImageNet release
- **torchvision:** `alexnet` (IMAGENET1K_V1): 61.1M params, 0.71 GFLOPS, 56.522% top-1, 79.066% top-5

```python
import torch, torch.nn as nn
from torchvision.models import alexnet, AlexNet_Weights

# PyTorch's LocalResponseNorm divides alpha by the window size n,
# so the paper's alpha = 1e-4 is alpha = 5e-4 here.
lrn = lambda: nn.LocalResponseNorm(size=5, alpha=5e-4, beta=0.75, k=2.0)

paper_alexnet = nn.Sequential(
    nn.Conv2d(3, 96, 11, stride=4), nn.ReLU(inplace=True), lrn(), nn.MaxPool2d(3, 2),            # 55 -> 27
    nn.Conv2d(96, 256, 5, padding=2, groups=2), nn.ReLU(inplace=True), lrn(), nn.MaxPool2d(3, 2),  # 27 -> 13
    nn.Conv2d(256, 384, 3, padding=1), nn.ReLU(inplace=True),
    nn.Conv2d(384, 384, 3, padding=1, groups=2), nn.ReLU(inplace=True),
    nn.Conv2d(384, 256, 3, padding=1, groups=2), nn.ReLU(inplace=True), nn.MaxPool2d(3, 2),       # 13 -> 6
    nn.Flatten(),
    nn.Dropout(0.5), nn.Linear(256 * 6 * 6, 4096), nn.ReLU(inplace=True),
    nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(inplace=True),
    nn.Linear(4096, 1000),
)
print(paper_alexnet(torch.randn(2, 3, 227, 227)).shape)        # torch.Size([2, 1000])
print(sum(p.numel() for p in paper_alexnet.parameters()))     # 60965224

tv = alexnet(weights=AlexNet_Weights.IMAGENET1K_V1).eval()    # 64-192-384-256-256, no LRN
print(sum(p.numel() for p in tv.parameters()))                # 61100840
preprocess = AlexNet_Weights.IMAGENET1K_V1.transforms()       # resize 256, centre-crop 224, normalise
```

The paper's two-GPU AlexNet as one module (groups=2), and torchvision's single-GPU variant

> **224 or 227, and LRN's alpha:** Two details trip up re-implementations. First, an $11\times11$ stride-4 conv only gives $55\times55$ from a $227\times227$ input, or from $224$ with padding 2; plain $224$ gives $54$ and every later shape is off by one. Second, `nn.LocalResponseNorm` uses $\alpha/n$ inside the sum, so copying the paper's $\alpha=10^{-4}$ makes the normalisation five times weaker than intended.

> **Parameters and compute live in different places:** 96% of AlexNet's parameters sit in three dense layers, while 92% of its multiply-adds happen in five conv layers. That is why the two problems got separate fixes: dropout for the dense layers' overfitting, and the GPU and the grouped convolutions for the conv layers' compute. Later networks attacked the dense layers directly: **GoogLeNet (Inception)** and **Residual neural network (ResNet)** replace them with global average pooling.

> **What survived:** ReLU, dropout, random-crop and flip augmentation, GPU training and the conv-then-classifier layout are still standard. LRN, the $11\times11$ stride-4 first layer and the two-GPU split did not survive. **ZFNet** shrank the first layer a year later, **VGGNet** dropped LRN, and grouped convolution came back in **ResNeXt** as a deliberate design choice.

**Strengths**

- Showed that a large CNN trained end to end beats hand-crafted features by a wide margin
- Simple, sequential layout that is easy to implement and to reason about
- Cheap by modern standards: 0.72 G multiply-adds per image
- Its training recipe (ReLU, dropout, augmentation, momentum SGD with weight decay) became the default

**Limitations**

- 96% of its 61M parameters are in the dense layers: large to store, prone to overfitting
- 11×11 stride-4 first layer aliases and wastes filters (ZFNet's diagnosis)
- LRN and the two-GPU split are hardware- and era-specific tricks
- Far less accurate than later nets of similar cost: torchvision's AlexNet reaches 56.5% top-1

**Question:** conv2 of AlexNet maps 96 input maps to 256 output maps with 5×5 kernels, split across two GPUs (groups = 2). How many parameters does it have?

- A. 307,456
- B. 614,656
- C. 153,728
- D. 6,400

**Answer:** A. 307,456 — Each kernel sees $96/2=48$ maps: $5\cdot5\cdot48\cdot256=307{,}200$ weights plus 256 biases is $307{,}456$. Without the split every kernel would see all 96 maps and the layer would have $614{,}656$ parameters.

**Question:** Where would dropout do the most good in AlexNet, and why did the authors put it there?

- A. In conv1, because it has the most activations
- B. In fc6 and fc7, because almost all the parameters are there
- C. After every LRN layer
- D. On the input pixels

**Answer:** B. In fc6 and fc7, because almost all the parameters are there — fc6, fc7 and fc8 hold 96% of the 61M parameters (fc6 alone has 37.8M), and that is where the network overfit. The conv layers have few parameters thanks to weight sharing and are regularised by augmentation instead.

**Key points**

- 5 conv + 3 dense layers, ≈ 61M parameters, 0.72 G multiply-adds; 227×227 input (the paper's "224" needs padding).
- ILSVRC-2012: 15.3% top-5 test error vs 26.2% for the runner-up, the result that turned vision to deep learning.
- ReLU (6× faster to train than tanh), dropout 0.5 in fc6/fc7, overlapping 3×3/2 max-pooling, LRN after conv1 and conv2.
- Augmentation: random 224 crops of 256 images plus flips (×2048), and PCA colour shifts with α ~ N(0, 0.1²).
- The two-GPU split is groups = 2 in conv2/4/5: it halves those layers' parameters and removes 36% of the compute of the same widths ungrouped.
- The dense layers hold 96% of the parameters; the conv layers do 92% of the compute.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
