Definition
LeNet-5 is the 7-layer convolutional network of LeCun, Bottou, Bengio and Haffner (1998). It maps a greyscale image through C1 (6 maps of ), S2 (subsampling to ), C3 (16 maps of , with a sparse connection table), S4 (), C5 (120 units), F6 (84 units) and 10 Euclidean RBF outputs, one per digit. Every layer except the output is trained end to end by backpropagation.
LeNet-5 is the network in "Gradient-based learning applied to document recognition" (Proceedings of the IEEE, 1998), the paper that also introduced the MNIST benchmark. It was the result of nearly a decade of work at AT&T Bell Labs. LeCun et al. (1989) had already trained a small convolutional net with backpropagation on US postal zip codes. LeNet-1 and LeNet-4 followed. The 1998 paper put the pieces together: local receptive fields, weight sharing and spatial subsampling, which together build some shift and distortion invariance into the architecture rather than hoping a fully connected net learns it.
It worked in production. The paper reports 0.95% test error on MNIST for LeNet-5, 0.8% when trained with artificially distorted images, and it describes a graph transformer network built on these convolutional recognisers that read bank cheques. That system was deployed commercially and read several million cheques a day. The network is tiny by today's standards: about 60,000 trainable parameters. Yet the stem of conv and pool stages, then dense layers, then a classifier, is the template that AlexNet, VGGNet and most later CNNs follow.
The layers and their arithmetic
For a convolution with stride 1 and no padding, an map becomes . Each output map has one kernel per input map it reads, plus one bias. A subsampling layer adds the four inputs of each non-overlapping window, multiplies the sum by one trainable coefficient per map, adds a trainable bias and squashes the result. In modern terms it is average pooling with a learnable scale.
- kernel side: 5 for C1, C3 and C5
- number of input maps output map reads: all of them in C1 and C5, 3, 4 or 6 in C3
- the subsampling layer's coefficient and bias: 2 parameters per map in S2 and S4
- the squashing function, , , chosen so and the slope at 0 is
The output layer is not a softmax. Each of the 10 units computes a squared Euclidean distance between the 84-dimensional F6 vector and a fixed prototype: . The prototypes were set by hand to bitmaps of each character drawn on a grid, which is why F6 has 84 units. The predicted class is the one with the smallest , and training lowers the distance to the correct prototype (the paper adds a term that also pushes the wrong classes' penalties up).
At a glance
- Paper
- LeCun, Bottou, Bengio, Haffner, Proc. IEEE 1998
- Input
- greyscale; MNIST's digits are padded so the digit box sits in the middle
- Layers
- C1 → S2 → C3 → S4 → C5 → F6 → 10 RBF outputs
- Parameters
- 60,000 trainable (+ 840 fixed RBF prototype values)
- Connections
- 340,908, so each weight is reused about 5.7 times on average
- Activation
- scaled tanh
- MNIST test error
- 0.95%; 0.8% with distorted training images (paper)
Why the C3 table is sparse
C3 does not read all six S2 maps. Its first six maps each read 3 adjacent maps, the next six read 4 adjacent maps, three read 4 non-adjacent maps, and the last one reads all six. The paper gives two reasons. It keeps the number of connections "within reasonable bounds", and, more interestingly, it breaks the symmetry of the network. Different C3 maps get different inputs, so they are forced to extract different, complementary features. Today the same job is done by random initialisation, and the sparse table is usually replaced by a full layer. Grouped convolutions (AlexNet's two-GPU split, ResNeXt) are its descendants: a structured way to have each output read only part of the input channels.
| layer | output | receptive field | trainable params | connections |
|---|---|---|---|---|
| C1: 6 kernels 5×5 | 6×28×28 | 5×5 | 6·(25+1) = 156 | 156·784 = 122,304 |
| S2: 2×2 subsample | 6×14×14 | 6×6 | 6·2 = 12 | 5,880 |
| C3: 16 kernels 5×5, Table I | 16×10×10 | 14×14 | 6·76 + 9·101 + 151 = 1,516 | 1,516·100 = 151,600 |
| S4: 2×2 subsample | 16×5×5 | 16×16 | 16·2 = 32 | 2,000 |
| C5: 120 kernels 5×5×16 | 120×1×1 | 32×32 | 120·(400+1) = 48,120 | 48,120 |
| F6: dense | 84 | all | 84·(120+1) = 10,164 | 10,164 |
| RBF output | 10 | all | 0 (840 fixed) | 840 |
| total | 60,000 | 340,908 |
C5 deserves a note. On a S4 map a kernel produces a single output, so C5 behaves like a dense layer. The paper still calls it a convolution on purpose: if the input were larger than , C5 would produce a spatial map and the whole network could slide over a bigger image. This is the earliest form of the "fully convolutional" trick that VGGNet later used for dense evaluation.
Worked example: the shapes and one C3 map
- 1C1: , so 6 maps of . Parameters . Each of the outputs uses 26 connections: .
- 2S2: . Each output uses 4 inputs, 1 coefficient and 1 bias. Connections , parameters .
- 3C3 map 13 reads S2 maps 1, 2, 4 and 5. Its parameters: . Its output is on a side, so it has connections.
- 4All of C3: six 3-input maps (), nine 4-input maps () and one 6-input map () give parameters. A full C3 would have .
- 5Receptive fields grow as , where is the product of the strides so far: (C1), (S2), (C3), (S4), (C5). A C5 unit sees the entire input.
1import torch, torch.nn as nn, torch.nn.functional as F23# Table I: which S2 maps each of the 16 C3 maps reads4C3_TABLE = [[0,1,2],[1,2,3],[2,3,4],[3,4,5],[4,5,0],[5,0,1],5 [0,1,2,3],[1,2,3,4],[2,3,4,5],[3,4,5,0],[4,5,0,1],[5,0,1,2],6 [0,1,3,4],[1,2,4,5],[0,2,3,5],[0,1,2,3,4,5]]78class MaskedConv2d(nn.Conv2d):9 def __init__(self, cin, cout, k, table):10 super().__init__(cin, cout, k)11 mask = torch.zeros(cout, cin, 1, 1)12 for o, ins in enumerate(table):13 mask[o, ins] = 114 self.register_buffer("mask", mask)15 def forward(self, x):16 return F.conv2d(x, self.weight * self.mask, self.bias)1718class LeNet5(nn.Module):19 def __init__(self, num_classes=10):20 super().__init__()21 act = lambda: nn.Tanh() # the paper uses 1.7159 * tanh(2a/3)22 self.features = nn.Sequential(23 nn.Conv2d(1, 6, 5), act(), # C1: 1x32x32 -> 6x28x2824 nn.AvgPool2d(2), # S2: -> 6x14x14 (paper: learnable scale + bias)25 MaskedConv2d(6, 16, 5, C3_TABLE), act(), # C3: -> 16x10x1026 nn.AvgPool2d(2), # S4: -> 16x5x527 nn.Conv2d(16, 120, 5), act(), # C5: -> 120x1x128 )29 self.classifier = nn.Sequential(30 nn.Flatten(), nn.Linear(120, 84), act(), # F631 nn.Linear(84, num_classes), # logits; use CrossEntropyLoss instead of the RBF layer32 )33 def forward(self, x):34 return self.classifier(self.features(x))3536net = LeNet5()37x = F.pad(torch.randn(8, 1, 28, 28), (2, 2, 2, 2)) # MNIST 28x28 -> 32x3238print(net(x).shape) # torch.Size([8, 10])39print(sum(p.numel() for p in net.parameters())) # 61706 stored (C3 stored densely)40c3 = net.features[3]41print(int(c3.mask.sum()) * 25 + 16) # 1516 effective C3 weights + biasesUse when
- +Very few parameters (60k) thanks to weight sharing
- +Built-in tolerance to small shifts and distortions from local fields and subsampling
- +Trains end to end on raw pixels: no hand-designed features
- +Fully convolutional up to C5, so it can slide over larger inputs
Avoid when
- −Saturating tanh units and hand-set RBF prototypes: harder to train than ReLU + softmax
- −Tiny capacity: fine for digits, far too small for natural images
- −Hand-designed C3 table: an architectural choice that random init made unnecessary
- −Subsampling by averaging blurs fine detail that max-pooling would keep
Check yourself
A C3 map in LeNet-5 reads 3 of the 6 S2 maps with 5×5 kernels. How many trainable parameters does that map have?
Check yourself
Why does the paper call C5 a convolutional layer even though its output is 1×1?
Remember
- ✓ C1 S2 C3 S4 C5 F6 10 RBF outputs.
- ✓60,000 trainable parameters cover 340,908 connections: weight sharing reuses each kernel at every position.
- ✓Subsampling = 2×2 sum × one trainable coefficient + bias per map, then a squash; today it is average or max pooling.
- ✓The C3 connection table limits connections and breaks symmetry: 1,516 parameters instead of 2,416.
- ✓Scaled tanh 1.7159·tanh(2a/3) hidden units; Euclidean RBF outputs with fixed prototypes instead of softmax.
- ✓0.95% MNIST test error (0.8% with distortions); deployed in cheque reading. Its conv-pool-dense layout is the template for AlexNet and VGG.