Neural network architecture

An architecture fixes the layers, their widths and how they connect; depth composes features, width adds capacity, and the parameter count follows from the shapes.

7 min read693 wordscore0 viewsupdated 2026-10-09

Definition

Architecture is the fixed blueprint of a network: which layers it has, how many units each layer has (its width), how many layers are stacked (its depth), and which outputs feed which inputs (its connectivity). Training chooses the values of the weights; the architecture chooses which weights exist.

Two networks trained on the same data with the same optimizer can behave very differently purely because of their shapes. The architecture sets how many parameters there are, what functions are cheap to represent, and what the model assumes about its input before it has seen any data.

Depth, width, connectivity

Depth is the number of layers with weights. Each layer re-describes the previous layer's output, so a deep network is a composition fL∘⋯∘f2∘f1f_L\circ\dots\circ f_2\circ f_1. Early layers can compute simple features and later layers combine them into more abstract ones (edges, then parts, then objects in a vision network).

Width is the number of units in a layer. More units means more features computed in parallel at that level, and more parameters. A single hidden layer that is wide enough can approximate any continuous function on a bounded domain (the universal approximation theorem), but "wide enough" can be enormous.

That is why depth matters in practice. There are functions that a deep network represents with a modest number of units while any shallow network needs exponentially many. Composition reuses intermediate features instead of re-deriving them from scratch.

Connectivity says which units talk to which. In a dense (fully connected) layer, every output unit sees every input unit. This article counts dense layers; the other families get their own topics later: convolutional layers connect each output only to a small local patch and reuse the same kernel everywhere, recurrent layers reuse one set of weights at every time step, and attention layers let every position compute data-dependent weights over every other position.

Shapes and parameter count

A dense layer maps a vector with ninn_{\text{in}} entries to one with noutn_{\text{out}} entries. Each output unit has one weight per input plus one bias, so the shapes are forced:

h(ℓ)=ϕ ⁣(W(ℓ)h(ℓ−1)+b(ℓ)),W(ℓ)∈Rnℓ×nℓ−1,b(ℓ)∈Rnℓh^{(\ell)}=\phi\!\left(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}\right),\qquad W^{(\ell)}\in\mathbb{R}^{n_\ell\times n_{\ell-1}},\quad b^{(\ell)}\in\mathbb{R}^{n_\ell}
h(ℓ)h^{(\ell)}
output vector of layer ℓ\ell; h(0)=xh^{(0)}=x is the input
nℓn_\ell
width of layer ℓ\ell (n0n_0 is the number of input features)
W(ℓ)W^{(\ell)}
weight matrix: one row per output unit, one column per input unit
b(ℓ)b^{(\ell)}
bias vector, one entry per output unit
ϕ\phi
elementwise activation, e.g. ReLU
Pℓ=nℓ−1 nℓ+nℓ,P=∑ℓ=1L(nℓ−1 nℓ+nℓ)P_\ell=n_{\ell-1}\,n_\ell+n_\ell,\qquad P=\sum_{\ell=1}^{L}\left(n_{\ell-1}\,n_\ell+n_\ell\right)
PℓP_\ell
trainable parameters of layer ℓ\ell (weights + biases)
LL
number of weight layers (the depth)
PP
total parameter count of the network

The input layer has no parameters: it just holds xx. Activations have no parameters either. Because PℓP_\ell is a product of adjacent widths, the count is dominated by the largest adjacent pair, which for image inputs is almost always the first layer.

W1: 128×784100,480W2: 64×1288,256W3: 10×64650⋮784input⋮128hidden 1⋮64hidden 2⋮10output
input784hidden 1128hidden 264output10
|
layerW shapeweightsbiasesparamsshare
input → hidden 1128 × 784100,352128100,480
hidden 1 → hidden 264 × 1288,192648,256
hidden 2 → output10 × 6464010650
total (3 weight matrices, depth 3)109,38691.9% in layer 1
fig.Build a dense network: change any layer's width with − / +, add or remove hidden layers, or pick a preset. The diagram redraws (layers wider than 8 are elided with ⋮) and the table gives each weight matrix's shape and n_in·n_out + n_out. Hover a row to highlight its edges. Compare the 784-128-64-10 and 784-138-10 presets: about the same budget, different depth.

Worked example: 784-128-64-10

A classic MNIST classifier flattens a 28×2828\times28 image into 784 inputs, uses hidden layers of 128 and 64 units, and outputs 10 class scores.

LayerW shapeWeightsBiasesParameters
input → hidden 1128 × 784100,352128100,480
hidden 1 → hidden 264 × 1288,192648,256
hidden 2 → output10 × 6464010650
total109,184202109,386
Parameter count of the 784-128-64-10 network

The first layer holds 91.9% of all parameters. A shallower network with the same budget, 784-138-10, has 108,330+1,390=109,720108{,}330+1{,}390=109{,}720 parameters, almost the same count, but only one stage of feature composition. Equal parameter counts do not mean equal behaviour.

python
1import torch.nn as nn
2
3net = nn.Sequential(
4 nn.Linear(784, 128), nn.ReLU(),
5 nn.Linear(128, 64), nn.ReLU(),
6 nn.Linear(64, 10),
7)
8print(net[0].weight.shape) # torch.Size([128, 784]) (out, in)
9print(sum(p.numel() for p in net.parameters())) # 109386

Inductive bias

Every architecture builds in assumptions about the data, called its inductive bias. A dense layer assumes almost nothing: shuffle the 784 pixels with a fixed permutation and an MLP learns just as well, because it never knew which pixels were neighbours.

That freedom is expensive. A dense map from a 28×2828\times28 image to 32 feature maps of the same size needs 784⋅784⋅32≈19.7784\cdot784\cdot32\approx19.7 million weights. A 3×33\times3 convolution producing the same 32 maps needs 3⋅3⋅32+32=3203\cdot3\cdot32+32=320 parameters, because it assumes locality and that a pattern means the same thing wherever it appears. When the assumption is true, the stronger bias generalizes from far less data.

Check yourself

How many trainable parameters does a dense layer from 256 inputs to 32 outputs have?

Check yourself

You double every hidden width of 784-128-64-10 to get 784-256-128-10. The parameter count roughly…

Remember

  • ✓An architecture fixes depth (how many layers), width (units per layer) and connectivity (who feeds whom).
  • ✓A dense layer has W∈Rnout×ninW\in\mathbb{R}^{n_{\text{out}}\times n_{\text{in}}} and b∈Rnoutb\in\mathbb{R}^{n_{\text{out}}}: ninnout+noutn_{\text{in}}n_{\text{out}}+n_{\text{out}} parameters.
  • ✓784-128-64-10 has 109,386 parameters, 92% of them in the first layer.
  • ✓Depth composes features; width adds capacity per level; equal budgets can behave differently.
  • ✓Conv, recurrent and attention layers trade generality for an inductive bias that fits the data.
← back to Deep Learning roadmap