# LeNet-5

> LeCun et al.'s 1998 digit recogniser: two conv + subsampling stages, then C5, F6 and an RBF output. About 60k trainable weights, shared across positions, cover 340,908 connections. It set the conv-pool-dense template every later CNN follows.

Source: https://www.mlfoundry.in/learn/lenet-5  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 12 min read · updated 2026-10-11

**LeNet-5:** is the 7-layer convolutional network of LeCun, Bottou, Bengio and Haffner (1998). It maps a $32\times32$ greyscale image through **C1** (6 maps of $28\times28$), **S2** (subsampling to $14\times14$), **C3** (16 maps of $10\times10$, with a sparse connection table), **S4** ($5\times5$), **C5** (120 units), **F6** (84 units) and 10 Euclidean RBF outputs, one per digit. Every layer except the output is trained end to end by backpropagation.

LeNet-5 is the network in "Gradient-based learning applied to document recognition" (Proceedings of the IEEE, 1998), the paper that also introduced the MNIST benchmark. It was the result of nearly a decade of work at AT&T Bell Labs. LeCun et al. (1989) had already trained a small convolutional net with backpropagation on US postal zip codes. LeNet-1 and LeNet-4 followed. The 1998 paper put the pieces together: local receptive fields, weight sharing and spatial subsampling, which together build some shift and distortion invariance into the architecture rather than hoping a fully connected net learns it.

It worked in production. The paper reports 0.95% test error on MNIST for LeNet-5, 0.8% when trained with artificially distorted images, and it describes a graph transformer network built on these convolutional recognisers that read bank cheques. That system was deployed commercially and read several million cheques a day. The network is tiny by today's standards: about 60,000 trainable parameters. Yet the stem of conv and pool stages, then dense layers, then a classifier, is the template that **AlexNet**, **VGGNet** and most later CNNs follow.

## The layers and their arithmetic

For a $k\times k$ convolution with stride 1 and no padding, an $n\times n$ map becomes $(n-k+1)\times(n-k+1)$. Each output map has one $k\times k$ kernel per input map it reads, plus one bias. A subsampling layer adds the four inputs of each non-overlapping $2\times2$ window, multiplies the sum by **one trainable coefficient** per map, adds a trainable bias and squashes the result. In modern terms it is average pooling with a learnable scale.

$$
n_{out}=n_{in}-k+1,\qquad P_{conv}=\sum_{j=1}^{C_{out}}\big(m_j\,k^2+1\big),\qquad y_{ij}=f\!\Big(\beta\sum_{(u,v)\in 2\times2}x_{2i+u,\,2j+v}+b\Big)
$$

- $k$: kernel side: 5 for C1, C3 and C5
- $m_j$: number of input maps output map $j$ reads: all of them in C1 and C5, 3, 4 or 6 in C3
- $\beta,\,b$: the subsampling layer's coefficient and bias: 2 parameters per map in S2 and S4
- $f(a)=A\tanh(Sa)$: the squashing function, $A=1.7159$, $S=2/3$, chosen so $f(\pm1)=\pm1$ and the slope at 0 is $AS=1.144$

The output layer is not a softmax. Each of the 10 units computes a squared Euclidean distance between the 84-dimensional F6 vector and a fixed prototype: $y_i=\sum_j (x_j-w_{ij})^2$. The prototypes were set by hand to $\pm1$ bitmaps of each character drawn on a $7\times12$ grid, which is why F6 has 84 units. The predicted class is the one with the **smallest** $y_i$, and training lowers the distance to the correct prototype (the paper adds a term that also pushes the wrong classes' penalties up).

**At a glance**

- **Paper:** LeCun, Bottou, Bengio, Haffner, Proc. IEEE 1998
- **Input:** $32\times32$ greyscale; MNIST's $28\times28$ digits are padded so the $20\times20$ digit box sits in the middle
- **Layers:** C1 → S2 → C3 → S4 → C5 → F6 → 10 RBF outputs
- **Parameters:** 60,000 trainable (+ 840 fixed RBF prototype values)
- **Connections:** 340,908, so each weight is reused about 5.7 times on average
- **Activation:** scaled tanh $1.7159\tanh(2a/3)$
- **MNIST test error:** 0.95%; 0.8% with distorted training images (paper)

> Figure: LeNet-5 as specified in the paper. Click a block to see its output shape, parameters, connections and receptive field. Click the digit to place a unit: the box marks the pixels it sees, 14×14 for a C3 unit and the whole 32×32 input for C5. The connection table on the right is Table I of the paper. The highlighted map 13 reads 4 of the 6 S2 maps, so it has 4·25+1 = 101 parameters, and all of C3 has 1,516. Switch C3 to fully connected: it grows to 2,416 parameters and the network gains 90,000 connections. The bars show the imbalance: C3 holds 2.5% of the parameters but 44.5% of the connections, and C5 holds 79% (48,120 of the 60,840 stored values, counting the fixed RBF prototypes). The modern re-implementation (average pooling, full C3, linear logits) has 61,706 parameters.

## Why the C3 table is sparse

C3 does not read all six S2 maps. Its first six maps each read 3 adjacent maps, the next six read 4 adjacent maps, three read 4 non-adjacent maps, and the last one reads all six. The paper gives two reasons. It keeps the number of connections "within reasonable bounds", and, more interestingly, it **breaks the symmetry** of the network. Different C3 maps get different inputs, so they are forced to extract different, complementary features. Today the same job is done by random initialisation, and the sparse table is usually replaced by a full layer. Grouped convolutions (AlexNet's two-GPU split, **ResNeXt**) are its descendants: a structured way to have each output read only part of the input channels.

Every layer of LeNet-5 (computed from the layer specs and checked against the totals the paper states)

| layer | output | receptive field | trainable params | connections |
| --- | --- | --- | --- | --- |
| C1: 6 kernels 5×5 | 6×28×28 | 5×5 | 6·(25+1) = 156 | 156·784 = 122,304 |
| S2: 2×2 subsample | 6×14×14 | 6×6 | 6·2 = 12 | 5,880 |
| C3: 16 kernels 5×5, Table I | 16×10×10 | 14×14 | 6·76 + 9·101 + 151 = 1,516 | 1,516·100 = 151,600 |
| S4: 2×2 subsample | 16×5×5 | 16×16 | 16·2 = 32 | 2,000 |
| C5: 120 kernels 5×5×16 | 120×1×1 | 32×32 | 120·(400+1) = 48,120 | 48,120 |
| F6: dense | 84 | all | 84·(120+1) = 10,164 | 10,164 |
| RBF output | 10 | all | 0 (840 fixed) | 840 |
| **total** |  |  | **60,000** | **340,908** |

C5 deserves a note. On a $5\times5$ S4 map a $5\times5$ kernel produces a single output, so C5 behaves like a dense layer. The paper still calls it a convolution on purpose: if the input were larger than $32\times32$, C5 would produce a spatial map and the whole network could slide over a bigger image. This is the earliest form of the "fully convolutional" trick that **VGGNet** later used for dense evaluation.

**Worked example: the shapes and one C3 map**

1. C1: $32-5+1=28$, so 6 maps of $28\times28$. Parameters $6\cdot(5\cdot5\cdot1+1)=156$. Each of the $6\cdot784=4{,}704$ outputs uses 26 connections: $122{,}304$.
2. S2: $28/2=14$. Each output uses 4 inputs, 1 coefficient and 1 bias. Connections $6\cdot196\cdot5=5{,}880$, parameters $12$.
3. C3 map 13 reads S2 maps 1, 2, 4 and 5. Its parameters: $4\cdot25+1=101$. Its output is $14-5+1=10$ on a side, so it has $100\cdot101=10{,}100$ connections.
4. All of C3: six 3-input maps ($6\cdot76$), nine 4-input maps ($9\cdot101$) and one 6-input map ($151$) give $456+909+151=1{,}516$ parameters. A full C3 would have $16\cdot(6\cdot25+1)=2{,}416$.
5. Receptive fields grow as $r\leftarrow r+(k-1)\cdot j$, where $j$ is the product of the strides so far: $5$ (C1), $6$ (S2), $5+(5-1)\cdot2+1=14$ (C3), $16$ (S4), $16+4\cdot4=32$ (C5). A C5 unit sees the entire input.

```python
import torch, torch.nn as nn, torch.nn.functional as F

# Table I: which S2 maps each of the 16 C3 maps reads
C3_TABLE = [[0,1,2],[1,2,3],[2,3,4],[3,4,5],[4,5,0],[5,0,1],
            [0,1,2,3],[1,2,3,4],[2,3,4,5],[3,4,5,0],[4,5,0,1],[5,0,1,2],
            [0,1,3,4],[1,2,4,5],[0,2,3,5],[0,1,2,3,4,5]]

class MaskedConv2d(nn.Conv2d):
    def __init__(self, cin, cout, k, table):
        super().__init__(cin, cout, k)
        mask = torch.zeros(cout, cin, 1, 1)
        for o, ins in enumerate(table):
            mask[o, ins] = 1
        self.register_buffer("mask", mask)
    def forward(self, x):
        return F.conv2d(x, self.weight * self.mask, self.bias)

class LeNet5(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        act = lambda: nn.Tanh()   # the paper uses 1.7159 * tanh(2a/3)
        self.features = nn.Sequential(
            nn.Conv2d(1, 6, 5), act(),           # C1: 1x32x32 -> 6x28x28
            nn.AvgPool2d(2),                     # S2: -> 6x14x14 (paper: learnable scale + bias)
            MaskedConv2d(6, 16, 5, C3_TABLE), act(),  # C3: -> 16x10x10
            nn.AvgPool2d(2),                     # S4: -> 16x5x5
            nn.Conv2d(16, 120, 5), act(),        # C5: -> 120x1x1
        )
        self.classifier = nn.Sequential(
            nn.Flatten(), nn.Linear(120, 84), act(),   # F6
            nn.Linear(84, num_classes),          # logits; use CrossEntropyLoss instead of the RBF layer
        )
    def forward(self, x):
        return self.classifier(self.features(x))

net = LeNet5()
x = F.pad(torch.randn(8, 1, 28, 28), (2, 2, 2, 2))   # MNIST 28x28 -> 32x32
print(net(x).shape)                                  # torch.Size([8, 10])
print(sum(p.numel() for p in net.parameters()))      # 61706 stored (C3 stored densely)
c3 = net.features[3]
print(int(c3.mask.sum()) * 25 + 16)                  # 1516 effective C3 weights + biases
```

A faithful-in-spirit LeNet-5 in PyTorch, with the C3 table as a weight mask

> **Most "LeNet-5" code is not LeNet-5:** Tutorials usually use max-pooling, ReLU, a fully connected C3 and a softmax output, often with $5\times5$ "same" padding on $28\times28$ inputs. That is a fine small CNN, with 61,706 parameters in the version above, but its numbers are not the paper's. When you quote the original, say 60,000 trainable parameters, 340,908 connections, trainable subsampling, the C3 table and Euclidean RBF outputs.

> **Weight sharing is the whole trick:** C1 has 122,304 connections but only 156 parameters, because the same six $5\times5$ kernels are applied at all 784 positions. A fully connected layer with the same connections would need 122,304 weights. Sharing is what makes the network cheap to store and data-efficient to train, and it makes C1's response to a stroke the same wherever the stroke appears. The cost moves from memory into computation, which is why compute, not parameter count, limited CNNs until GPUs arrived.

> **Where the parameters live:** Already in 1998 most parameters sit at the top: C5 alone has 48,120 of the 60,000 (80%), while C1 and C3 together have under 3%. The same imbalance appears in **AlexNet** (96% in the dense layers) and **VGGNet** (fc6 alone is 74% of VGG-16). Global average pooling in later networks removed most of it.

**Strengths**

- Very few parameters (60k) thanks to weight sharing
- Built-in tolerance to small shifts and distortions from local fields and subsampling
- Trains end to end on raw pixels: no hand-designed features
- Fully convolutional up to C5, so it can slide over larger inputs

**Limitations**

- Saturating tanh units and hand-set RBF prototypes: harder to train than ReLU + softmax
- Tiny capacity: fine for digits, far too small for natural images
- Hand-designed C3 table: an architectural choice that random init made unnecessary
- Subsampling by averaging blurs fine detail that max-pooling would keep

**Question:** A C3 map in LeNet-5 reads 3 of the 6 S2 maps with 5×5 kernels. How many trainable parameters does that map have?

- A. 25
- B. 76
- C. 151
- D. 7,600

**Answer:** B. 76 — One $5\times5$ kernel per input map plus one bias: $3\cdot25+1=76$. Six of C3's maps are like this. 7,600 is its number of connections ($76$ weights times 100 output positions).

**Question:** Why does the paper call C5 a convolutional layer even though its output is 1×1?

- A. Because its kernels are shared between the 120 outputs
- B. Because it has no bias
- C. Because with a larger input image it would produce a spatial map, so the net can slide over the image
- D. Because it uses the RBF function

**Answer:** C. Because with a larger input image it would produce a spatial map, so the net can slide over the image — On the $5\times5$ S4 map a $5\times5$ kernel has exactly one valid position, so C5 acts like a dense layer. Feed a larger image and it yields a grid of 120-vectors, one per $32\times32$ window. The network becomes a sliding-window detector for free.

**Key points**

- $32\times32\to$ C1 $6@28^2\to$ S2 $6@14^2\to$ C3 $16@10^2\to$ S4 $16@5^2\to$ C5 $120\to$ F6 $84\to$ 10 RBF outputs.
- 60,000 trainable parameters cover 340,908 connections: weight sharing reuses each kernel at every position.
- Subsampling = 2×2 sum × one trainable coefficient + bias per map, then a squash; today it is average or max pooling.
- The C3 connection table limits connections and breaks symmetry: 1,516 parameters instead of 2,416.
- Scaled tanh 1.7159·tanh(2a/3) hidden units; Euclidean RBF outputs with fixed prototypes instead of softmax.
- 0.95% MNIST test error (0.8% with distortions); deployed in cheque reading. Its conv-pool-dense layout is the template for AlexNet and VGG.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
