# Neural network architecture

> An architecture fixes the layers, their widths and how they connect; depth composes features, width adds capacity, and the parameter count follows from the shapes.

Source: https://www.mlfoundry.in/learn/neural-network-architecture  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Architecture:** is the fixed blueprint of a network: which layers it has, how many units each layer has (its **width**), how many layers are stacked (its **depth**), and which outputs feed which inputs (its **connectivity**). Training chooses the values of the weights; the architecture chooses which weights exist.

Two networks trained on the same data with the same optimizer can behave very differently purely because of their shapes. The architecture sets how many parameters there are, what functions are cheap to represent, and what the model assumes about its input before it has seen any data.

## Depth, width, connectivity

**Depth** is the number of layers with weights. Each layer re-describes the previous layer's output, so a deep network is a composition $f_L\circ\dots\circ f_2\circ f_1$. Early layers can compute simple features and later layers combine them into more abstract ones (edges, then parts, then objects in a vision network).

**Width** is the number of units in a layer. More units means more features computed in parallel at that level, and more parameters. A single hidden layer that is wide enough can approximate any continuous function on a bounded domain (the universal approximation theorem), but "wide enough" can be enormous.

That is why depth matters in practice. There are functions that a deep network represents with a modest number of units while any shallow network needs exponentially many. Composition reuses intermediate features instead of re-deriving them from scratch.

**Connectivity** says which units talk to which. In a **dense** (fully connected) layer, every output unit sees every input unit. This article counts dense layers; the other families get their own topics later: **convolutional** layers connect each output only to a small local patch and reuse the same kernel everywhere, **recurrent** layers reuse one set of weights at every time step, and **attention** layers let every position compute data-dependent weights over every other position.

## Shapes and parameter count

A dense layer maps a vector with $n_{\text{in}}$ entries to one with $n_{\text{out}}$ entries. Each output unit has one weight per input plus one bias, so the shapes are forced:

$$
h^{(\ell)}=\phi\!\left(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}\right),\qquad W^{(\ell)}\in\mathbb{R}^{n_\ell\times n_{\ell-1}},\quad b^{(\ell)}\in\mathbb{R}^{n_\ell}
$$

- $h^{(\ell)}$: output vector of layer $\ell$; $h^{(0)}=x$ is the input
- $n_\ell$: width of layer $\ell$ ($n_0$ is the number of input features)
- $W^{(\ell)}$: weight matrix: one row per output unit, one column per input unit
- $b^{(\ell)}$: bias vector, one entry per output unit
- $\phi$: elementwise activation, e.g. ReLU

$$
P_\ell=n_{\ell-1}\,n_\ell+n_\ell,\qquad P=\sum_{\ell=1}^{L}\left(n_{\ell-1}\,n_\ell+n_\ell\right)
$$

- $P_\ell$: trainable parameters of layer $\ell$ (weights + biases)
- $L$: number of weight layers (the depth)
- $P$: total parameter count of the network

The input layer has no parameters: it just holds $x$. Activations have no parameters either. Because $P_\ell$ is a product of adjacent widths, the count is dominated by the largest adjacent pair, which for image inputs is almost always the first layer.

> Figure: Build a dense network: change any layer's width with − / +, add or remove hidden layers, or pick a preset. The diagram redraws (layers wider than 8 are elided with ⋮) and the table gives each weight matrix's shape and n_in·n_out + n_out. Hover a row to highlight its edges. Compare the 784-128-64-10 and 784-138-10 presets: about the same budget, different depth.

## Worked example: 784-128-64-10

A classic MNIST classifier flattens a $28\times28$ image into 784 inputs, uses hidden layers of 128 and 64 units, and outputs 10 class scores.

Parameter count of the 784-128-64-10 network

| Layer | W shape | Weights | Biases | Parameters |
| --- | --- | --- | --- | --- |
| input → hidden 1 | 128 × 784 | 100,352 | 128 | 100,480 |
| hidden 1 → hidden 2 | 64 × 128 | 8,192 | 64 | 8,256 |
| hidden 2 → output | 10 × 64 | 640 | 10 | 650 |
| **total** |  | 109,184 | 202 | **109,386** |

The first layer holds 91.9% of all parameters. A shallower network with the same budget, 784-138-10, has $108{,}330+1{,}390=109{,}720$ parameters, almost the same count, but only one stage of feature composition. Equal parameter counts do not mean equal behaviour.

```python
import torch.nn as nn

net = nn.Sequential(
    nn.Linear(784, 128), nn.ReLU(),
    nn.Linear(128, 64), nn.ReLU(),
    nn.Linear(64, 10),
)
print(net[0].weight.shape)                       # torch.Size([128, 784])  (out, in)
print(sum(p.numel() for p in net.parameters()))  # 109386
```

> **PyTorch stores W as (out, in):** `nn.Linear(784, 128)` is called with (in, out), but its `weight` has shape `(128, 784)`, matching $W\in\mathbb{R}^{n_{\text{out}}\times n_{\text{in}}}$ and computing $xW^\top+b$ on a batch of row vectors. Mixing up the two orders is the most common source of shape errors when porting weights.

## Inductive bias

Every architecture builds in assumptions about the data, called its **inductive bias**. A dense layer assumes almost nothing: shuffle the 784 pixels with a fixed permutation and an MLP learns just as well, because it never knew which pixels were neighbours.

That freedom is expensive. A dense map from a $28\times28$ image to 32 feature maps of the same size needs $784\cdot784\cdot32\approx19.7$ million weights. A $3\times3$ convolution producing the same 32 maps needs $3\cdot3\cdot32+32=320$ parameters, because it assumes locality and that a pattern means the same thing wherever it appears. When the assumption is true, the stronger bias generalizes from far less data.

> **Architecture is a prior:** Choosing an architecture is choosing which functions are easy to learn. Convolutions favour local, translation-equivariant features; recurrence favours sequential processing with shared weights over time; attention favours content-based interactions between all pairs of positions. The best architecture matches the structure of the data.

**Question:** How many trainable parameters does a dense layer from 256 inputs to 32 outputs have?

- A. 8,192
- B. 8,224
- C. 8,448
- D. 288

**Answer:** B. 8,224 — $256\cdot32=8{,}192$ weights plus $32$ biases $=8{,}224$. (8,448 would wrongly use 256 biases; 288 adds the widths instead of multiplying them.)

**Question:** You double every hidden width of 784-128-64-10 to get 784-256-128-10. The parameter count roughly…

- A. stays the same
- B. doubles
- C. more than doubles, because hidden-to-hidden terms grow 4×
- D. quadruples exactly

**Answer:** C. more than doubles, because hidden-to-hidden terms grow 4× — Terms between two hidden layers scale as $2n\cdot2m=4nm$, while input- and output-adjacent terms only double. Here: $200{,}960+32{,}896+1{,}290=235{,}146$, about $2.15\times$ the original 109,386.

**Key points**

- An architecture fixes depth (how many layers), width (units per layer) and connectivity (who feeds whom).
- A dense layer has $W\in\mathbb{R}^{n_{\text{out}}\times n_{\text{in}}}$ and $b\in\mathbb{R}^{n_{\text{out}}}$: $n_{\text{in}}n_{\text{out}}+n_{\text{out}}$ parameters.
- 784-128-64-10 has 109,386 parameters, 92% of them in the first layer.
- Depth composes features; width adds capacity per level; equal budgets can behave differently.
- Conv, recurrent and attention layers trade generality for an inductive bias that fits the data.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
