Definition
Architecture is the fixed blueprint of a network: which layers it has, how many units each layer has (its width), how many layers are stacked (its depth), and which outputs feed which inputs (its connectivity). Training chooses the values of the weights; the architecture chooses which weights exist.
Two networks trained on the same data with the same optimizer can behave very differently purely because of their shapes. The architecture sets how many parameters there are, what functions are cheap to represent, and what the model assumes about its input before it has seen any data.
Depth, width, connectivity
Depth is the number of layers with weights. Each layer re-describes the previous layer's output, so a deep network is a composition . Early layers can compute simple features and later layers combine them into more abstract ones (edges, then parts, then objects in a vision network).
Width is the number of units in a layer. More units means more features computed in parallel at that level, and more parameters. A single hidden layer that is wide enough can approximate any continuous function on a bounded domain (the universal approximation theorem), but "wide enough" can be enormous.
That is why depth matters in practice. There are functions that a deep network represents with a modest number of units while any shallow network needs exponentially many. Composition reuses intermediate features instead of re-deriving them from scratch.
Connectivity says which units talk to which. In a dense (fully connected) layer, every output unit sees every input unit. This article counts dense layers; the other families get their own topics later: convolutional layers connect each output only to a small local patch and reuse the same kernel everywhere, recurrent layers reuse one set of weights at every time step, and attention layers let every position compute data-dependent weights over every other position.
Shapes and parameter count
A dense layer maps a vector with entries to one with entries. Each output unit has one weight per input plus one bias, so the shapes are forced:
- output vector of layer ; is the input
- width of layer ( is the number of input features)
- weight matrix: one row per output unit, one column per input unit
- bias vector, one entry per output unit
- elementwise activation, e.g. ReLU
- trainable parameters of layer (weights + biases)
- number of weight layers (the depth)
- total parameter count of the network
The input layer has no parameters: it just holds . Activations have no parameters either. Because is a product of adjacent widths, the count is dominated by the largest adjacent pair, which for image inputs is almost always the first layer.
| layer | W shape | weights | biases | params | share |
|---|---|---|---|---|---|
| input → hidden 1 | 128 × 784 | 100,352 | 128 | 100,480 | |
| hidden 1 → hidden 2 | 64 × 128 | 8,192 | 64 | 8,256 | |
| hidden 2 → output | 10 × 64 | 640 | 10 | 650 | |
| total (3 weight matrices, depth 3) | 109,386 | 91.9% in layer 1 | |||
Worked example: 784-128-64-10
A classic MNIST classifier flattens a image into 784 inputs, uses hidden layers of 128 and 64 units, and outputs 10 class scores.
| Layer | W shape | Weights | Biases | Parameters |
|---|---|---|---|---|
| input → hidden 1 | 128 × 784 | 100,352 | 128 | 100,480 |
| hidden 1 → hidden 2 | 64 × 128 | 8,192 | 64 | 8,256 |
| hidden 2 → output | 10 × 64 | 640 | 10 | 650 |
| total | 109,184 | 202 | 109,386 |
The first layer holds 91.9% of all parameters. A shallower network with the same budget, 784-138-10, has parameters, almost the same count, but only one stage of feature composition. Equal parameter counts do not mean equal behaviour.
1import torch.nn as nn23net = nn.Sequential(4 nn.Linear(784, 128), nn.ReLU(),5 nn.Linear(128, 64), nn.ReLU(),6 nn.Linear(64, 10),7)8print(net[0].weight.shape) # torch.Size([128, 784]) (out, in)9print(sum(p.numel() for p in net.parameters())) # 109386Inductive bias
Every architecture builds in assumptions about the data, called its inductive bias. A dense layer assumes almost nothing: shuffle the 784 pixels with a fixed permutation and an MLP learns just as well, because it never knew which pixels were neighbours.
That freedom is expensive. A dense map from a image to 32 feature maps of the same size needs million weights. A convolution producing the same 32 maps needs parameters, because it assumes locality and that a pattern means the same thing wherever it appears. When the assumption is true, the stronger bias generalizes from far less data.
Check yourself
How many trainable parameters does a dense layer from 256 inputs to 32 outputs have?
Check yourself
You double every hidden width of 784-128-64-10 to get 784-256-128-10. The parameter count roughly…
Remember
- ✓An architecture fixes depth (how many layers), width (units per layer) and connectivity (who feeds whom).
- ✓A dense layer has and : parameters.
- ✓784-128-64-10 has 109,386 parameters, 92% of them in the first layer.
- ✓Depth composes features; width adds capacity per level; equal budgets can behave differently.
- ✓Conv, recurrent and attention layers trade generality for an inductive bias that fits the data.