LeNet-5

LeCun et al.'s 1998 digit recogniser: two conv + subsampling stages, then C5, F6 and an RBF output. About 60k trainable weights, shared across positions, cover 340,908 connections. It set the conv-pool-dense template every later CNN follows.

12 min read1006 wordscore0 viewsupdated 2026-10-11

Definition

LeNet-5 is the 7-layer convolutional network of LeCun, Bottou, Bengio and Haffner (1998). It maps a 32×3232\times32 greyscale image through C1 (6 maps of 28×2828\times28), S2 (subsampling to 14×1414\times14), C3 (16 maps of 10×1010\times10, with a sparse connection table), S4 (5×55\times5), C5 (120 units), F6 (84 units) and 10 Euclidean RBF outputs, one per digit. Every layer except the output is trained end to end by backpropagation.

LeNet-5 is the network in "Gradient-based learning applied to document recognition" (Proceedings of the IEEE, 1998), the paper that also introduced the MNIST benchmark. It was the result of nearly a decade of work at AT&T Bell Labs. LeCun et al. (1989) had already trained a small convolutional net with backpropagation on US postal zip codes. LeNet-1 and LeNet-4 followed. The 1998 paper put the pieces together: local receptive fields, weight sharing and spatial subsampling, which together build some shift and distortion invariance into the architecture rather than hoping a fully connected net learns it.

It worked in production. The paper reports 0.95% test error on MNIST for LeNet-5, 0.8% when trained with artificially distorted images, and it describes a graph transformer network built on these convolutional recognisers that read bank cheques. That system was deployed commercially and read several million cheques a day. The network is tiny by today's standards: about 60,000 trainable parameters. Yet the stem of conv and pool stages, then dense layers, then a classifier, is the template that AlexNet, VGGNet and most later CNNs follow.

The layers and their arithmetic

For a k×kk\times k convolution with stride 1 and no padding, an n×nn\times n map becomes (n−k+1)×(n−k+1)(n-k+1)\times(n-k+1). Each output map has one k×kk\times k kernel per input map it reads, plus one bias. A subsampling layer adds the four inputs of each non-overlapping 2×22\times2 window, multiplies the sum by one trainable coefficient per map, adds a trainable bias and squashes the result. In modern terms it is average pooling with a learnable scale.

nout=nin−k+1,Pconv=∑j=1Cout(mj k2+1),yij=f ⁣(β∑(u,v)∈2×2x2i+u, 2j+v+b)n_{out}=n_{in}-k+1,\qquad P_{conv}=\sum_{j=1}^{C_{out}}\big(m_j\,k^2+1\big),\qquad y_{ij}=f\!\Big(\beta\sum_{(u,v)\in 2\times2}x_{2i+u,\,2j+v}+b\Big)
kk
kernel side: 5 for C1, C3 and C5
mjm_j
number of input maps output map jj reads: all of them in C1 and C5, 3, 4 or 6 in C3
β, b\beta,\,b
the subsampling layer's coefficient and bias: 2 parameters per map in S2 and S4
f(a)=Atanh⁡(Sa)f(a)=A\tanh(Sa)
the squashing function, A=1.7159A=1.7159, S=2/3S=2/3, chosen so f(±1)=±1f(\pm1)=\pm1 and the slope at 0 is AS=1.144AS=1.144

The output layer is not a softmax. Each of the 10 units computes a squared Euclidean distance between the 84-dimensional F6 vector and a fixed prototype: yi=∑j(xj−wij)2y_i=\sum_j (x_j-w_{ij})^2. The prototypes were set by hand to ±1\pm1 bitmaps of each character drawn on a 7×127\times12 grid, which is why F6 has 84 units. The predicted class is the one with the smallest yiy_i, and training lowers the distance to the correct prototype (the paper adds a term that also pushes the wrong classes' penalties up).

At a glance

Paper
LeCun, Bottou, Bengio, Haffner, Proc. IEEE 1998
Input
32×3232\times32 greyscale; MNIST's 28×2828\times28 digits are padded so the 20×2020\times20 digit box sits in the middle
Layers
C1 → S2 → C3 → S4 → C5 → F6 → 10 RBF outputs
Parameters
60,000 trainable (+ 840 fixed RBF prototype values)
Connections
340,908, so each weight is reused about 5.7 times on average
Activation
scaled tanh 1.7159tanh⁡(2a/3)1.7159\tanh(2a/3)
MNIST test error
0.95%; 0.8% with distorted training images (paper)
LeNet-5 (LeCun et al. 1998) · click a layer60,000 trainable · 340,908 conn.32²·1inC128²6S214²6C310²16S45²16C51120F6184RBF110input 32×32 · click a pixelC3 unit (3,6) sees rows 6–19layerconv 5×5, 16 maps, s1 p0in → out6×14×14 → 16×10×10params1,516connections151,600recept. field14×14 px, stride 2Table I: S2 maps (rows) → C3 maps0123456789101112131415S2·0S2·1S2·2S2·3S2·4S2·5map 13 ← 4 maps: 4·25+1 = 101C3 now: 1,516 params, 151,600 conn.table 1,516 vs full 2,416 paramstable saves 90,000 connectionsshare of parameters and connections (C3: 2.5% of params, 44.5% of conn.)paramsconn.■ conv ■ subsampling ■ dense / RBF · total params 60.8k
C3variant
layer
fig.LeNet-5 as specified in the paper. Click a block to see its output shape, parameters, connections and receptive field. Click the digit to place a unit: the box marks the pixels it sees, 14×14 for a C3 unit and the whole 32×32 input for C5. The connection table on the right is Table I of the paper. The highlighted map 13 reads 4 of the 6 S2 maps, so it has 4·25+1 = 101 parameters, and all of C3 has 1,516. Switch C3 to fully connected: it grows to 2,416 parameters and the network gains 90,000 connections. The bars show the imbalance: C3 holds 2.5% of the parameters but 44.5% of the connections, and C5 holds 79% (48,120 of the 60,840 stored values, counting the fixed RBF prototypes). The modern re-implementation (average pooling, full C3, linear logits) has 61,706 parameters.

Why the C3 table is sparse

C3 does not read all six S2 maps. Its first six maps each read 3 adjacent maps, the next six read 4 adjacent maps, three read 4 non-adjacent maps, and the last one reads all six. The paper gives two reasons. It keeps the number of connections "within reasonable bounds", and, more interestingly, it breaks the symmetry of the network. Different C3 maps get different inputs, so they are forced to extract different, complementary features. Today the same job is done by random initialisation, and the sparse table is usually replaced by a full layer. Grouped convolutions (AlexNet's two-GPU split, ResNeXt) are its descendants: a structured way to have each output read only part of the input channels.

layeroutputreceptive fieldtrainable paramsconnections
C1: 6 kernels 5×56×28×285×56·(25+1) = 156156·784 = 122,304
S2: 2×2 subsample6×14×146×66·2 = 125,880
C3: 16 kernels 5×5, Table I16×10×1014×146·76 + 9·101 + 151 = 1,5161,516·100 = 151,600
S4: 2×2 subsample16×5×516×1616·2 = 322,000
C5: 120 kernels 5×5×16120×1×132×32120·(400+1) = 48,12048,120
F6: dense84all84·(120+1) = 10,16410,164
RBF output10all0 (840 fixed)840
total60,000340,908
Every layer of LeNet-5 (computed from the layer specs and checked against the totals the paper states)

C5 deserves a note. On a 5×55\times5 S4 map a 5×55\times5 kernel produces a single output, so C5 behaves like a dense layer. The paper still calls it a convolution on purpose: if the input were larger than 32×3232\times32, C5 would produce a spatial map and the whole network could slide over a bigger image. This is the earliest form of the "fully convolutional" trick that VGGNet later used for dense evaluation.

Worked example: the shapes and one C3 map

  1. 1C1: 32−5+1=2832-5+1=28, so 6 maps of 28×2828\times28. Parameters 6⋅(5⋅5⋅1+1)=1566\cdot(5\cdot5\cdot1+1)=156. Each of the 6⋅784=4,7046\cdot784=4{,}704 outputs uses 26 connections: 122,304122{,}304.
  2. 2S2: 28/2=1428/2=14. Each output uses 4 inputs, 1 coefficient and 1 bias. Connections 6⋅196⋅5=5,8806\cdot196\cdot5=5{,}880, parameters 1212.
  3. 3C3 map 13 reads S2 maps 1, 2, 4 and 5. Its parameters: 4⋅25+1=1014\cdot25+1=101. Its output is 14−5+1=1014-5+1=10 on a side, so it has 100⋅101=10,100100\cdot101=10{,}100 connections.
  4. 4All of C3: six 3-input maps (6⋅766\cdot76), nine 4-input maps (9⋅1019\cdot101) and one 6-input map (151151) give 456+909+151=1,516456+909+151=1{,}516 parameters. A full C3 would have 16⋅(6⋅25+1)=2,41616\cdot(6\cdot25+1)=2{,}416.
  5. 5Receptive fields grow as r←r+(k−1)⋅jr\leftarrow r+(k-1)\cdot j, where jj is the product of the strides so far: 55 (C1), 66 (S2), 5+(5−1)⋅2+1=145+(5-1)\cdot2+1=14 (C3), 1616 (S4), 16+4⋅4=3216+4\cdot4=32 (C5). A C5 unit sees the entire input.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3# Table I: which S2 maps each of the 16 C3 maps reads
4C3_TABLE = [[0,1,2],[1,2,3],[2,3,4],[3,4,5],[4,5,0],[5,0,1],
5 [0,1,2,3],[1,2,3,4],[2,3,4,5],[3,4,5,0],[4,5,0,1],[5,0,1,2],
6 [0,1,3,4],[1,2,4,5],[0,2,3,5],[0,1,2,3,4,5]]
7
8class MaskedConv2d(nn.Conv2d):
9 def __init__(self, cin, cout, k, table):
10 super().__init__(cin, cout, k)
11 mask = torch.zeros(cout, cin, 1, 1)
12 for o, ins in enumerate(table):
13 mask[o, ins] = 1
14 self.register_buffer("mask", mask)
15 def forward(self, x):
16 return F.conv2d(x, self.weight * self.mask, self.bias)
17
18class LeNet5(nn.Module):
19 def __init__(self, num_classes=10):
20 super().__init__()
21 act = lambda: nn.Tanh() # the paper uses 1.7159 * tanh(2a/3)
22 self.features = nn.Sequential(
23 nn.Conv2d(1, 6, 5), act(), # C1: 1x32x32 -> 6x28x28
24 nn.AvgPool2d(2), # S2: -> 6x14x14 (paper: learnable scale + bias)
25 MaskedConv2d(6, 16, 5, C3_TABLE), act(), # C3: -> 16x10x10
26 nn.AvgPool2d(2), # S4: -> 16x5x5
27 nn.Conv2d(16, 120, 5), act(), # C5: -> 120x1x1
28 )
29 self.classifier = nn.Sequential(
30 nn.Flatten(), nn.Linear(120, 84), act(), # F6
31 nn.Linear(84, num_classes), # logits; use CrossEntropyLoss instead of the RBF layer
32 )
33 def forward(self, x):
34 return self.classifier(self.features(x))
35
36net = LeNet5()
37x = F.pad(torch.randn(8, 1, 28, 28), (2, 2, 2, 2)) # MNIST 28x28 -> 32x32
38print(net(x).shape) # torch.Size([8, 10])
39print(sum(p.numel() for p in net.parameters())) # 61706 stored (C3 stored densely)
40c3 = net.features[3]
41print(int(c3.mask.sum()) * 25 + 16) # 1516 effective C3 weights + biases
A faithful-in-spirit LeNet-5 in PyTorch, with the C3 table as a weight mask

Use when

  • +Very few parameters (60k) thanks to weight sharing
  • +Built-in tolerance to small shifts and distortions from local fields and subsampling
  • +Trains end to end on raw pixels: no hand-designed features
  • +Fully convolutional up to C5, so it can slide over larger inputs

Avoid when

  • −Saturating tanh units and hand-set RBF prototypes: harder to train than ReLU + softmax
  • −Tiny capacity: fine for digits, far too small for natural images
  • −Hand-designed C3 table: an architectural choice that random init made unnecessary
  • −Subsampling by averaging blurs fine detail that max-pooling would keep

Check yourself

A C3 map in LeNet-5 reads 3 of the 6 S2 maps with 5×5 kernels. How many trainable parameters does that map have?

Check yourself

Why does the paper call C5 a convolutional layer even though its output is 1×1?

Remember

  • ✓32×32→32\times32\to C1 6@282→6@28^2\to S2 6@142→6@14^2\to C3 16@102→16@10^2\to S4 16@52→16@5^2\to C5 120→120\to F6 84→84\to 10 RBF outputs.
  • ✓60,000 trainable parameters cover 340,908 connections: weight sharing reuses each kernel at every position.
  • ✓Subsampling = 2×2 sum × one trainable coefficient + bias per map, then a squash; today it is average or max pooling.
  • ✓The C3 connection table limits connections and breaks symmetry: 1,516 parameters instead of 2,416.
  • ✓Scaled tanh 1.7159·tanh(2a/3) hidden units; Euclidean RBF outputs with fixed prototypes instead of softmax.
  • ✓0.95% MNIST test error (0.8% with distortions); deployed in cheque reading. Its conv-pool-dense layout is the template for AlexNet and VGG.
← back to Deep Learning roadmap