Forward propagation

Compute each layer from the previous one, z = Wa + b then a = φ(z), until the output and the loss come out the end.

6 min read519 wordscore0 viewsupdated 2026-10-09

Definition

Forward propagation is the evaluation of a network on an input: starting from a(0)=xa^{(0)}=x, each layer applies an affine map followed by a nonlinearity to the previous layer's output, and the last layer's output is compared with the target to give a scalar loss.

A feed-forward network is a composition of functions, f(x)=f(L)(⋯f(2)(f(1)(x)))f(x)=f^{(L)}\big(\cdots f^{(2)}(f^{(1)}(x))\big). Forward propagation evaluates that composition from the inside out, one layer at a time. Nothing is learned during this pass; the weights are fixed.

It is used in two places. At inference it is the whole job: the output is the prediction. During training it is the first half of every step, and it must leave behind the intermediate values that the backward pass needs.

The layer recurrence

a(0)=x,z(l)=W(l)a(l−1)+b(l),a(l)=φ(l) ⁣(z(l)),l=1,…,La^{(0)}=x,\qquad z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=\varphi^{(l)}\!\big(z^{(l)}\big),\qquad l=1,\dots,L
a(l)∈Rnla^{(l)}\in\mathbb{R}^{n_l}
activations (outputs) of layer ll; nln_l is its width
W(l)∈Rnl×nl−1W^{(l)}\in\mathbb{R}^{n_l\times n_{l-1}}
weight matrix; row jj holds the incoming weights of neuron jj
b(l)∈Rnlb^{(l)}\in\mathbb{R}^{n_l}
bias vector, one entry per neuron
z(l)z^{(l)}
pre-activation: the weighted sum before the nonlinearity
φ(l)\varphi^{(l)}
activation function, applied elementwise (ReLU, sigmoid, …)

The output y^=a(L)\hat y=a^{(L)} then goes into the loss, L=ℓ(y^,y)\mathcal{L}=\ell(\hat y, y). For a binary classifier with a sigmoid output, that is the cross-entropy ℓ=−[ylog⁡y^+(1−y)log⁡(1−y^)]\ell=-[y\log\hat y+(1-y)\log(1-\hat y)].

Shapes are the best bug detector. W(l)W^{(l)} maps nl−1n_{l-1} numbers to nln_l numbers, so it is nl×nl−1n_l\times n_{l-1} (out × in). A layer from 784 inputs to 256 units has a 256×784256\times784 weight matrix and 256 biases: 200,960200{,}960 parameters.

Batched form

In practice a whole mini-batch goes through at once. Stack the BB examples as rows of a matrix, and each layer becomes one matrix multiply:

Z(l)=A(l−1) W(l)⊤+1 b(l)⊤,A(l)=φ(Z(l))Z^{(l)}=A^{(l-1)}\,W^{(l)\top}+\mathbf{1}\,b^{(l)\top},\qquad A^{(l)}=\varphi\big(Z^{(l)}\big)
A(l−1)∈RB×nl−1A^{(l-1)}\in\mathbb{R}^{B\times n_{l-1}}
previous layer's activations, one row per example
Z(l)∈RB×nlZ^{(l)}\in\mathbb{R}^{B\times n_l}
pre-activations for the whole batch
1 b(l)⊤\mathbf{1}\,b^{(l)\top}
the bias added to every row (broadcasting)

This is exactly what PyTorch's nn.Linear computes: it stores weight with shape (out, in) and returns x @ weight.T + bias. Each layer costs B nl nl−1B\,n_l\,n_{l-1} multiply-adds, which is why the forward pass is dominated by matrix multiplication and runs so well on GPUs.

step 0/4 · input: a⁽⁰⁾ = xlayer 0 (input)layer 1 (σ)layer 2 (σ)lossx10.50x21.00h1?b 0.1h2?b −0.1ŷ?b 0.2y = 1?
a⁽⁰⁾ = x = (0.50, 1.00)
cache for backprop:x = (0.50, 1.00)z⁽¹⁾ = …a⁽¹⁾ = …z⁽²⁾, ŷ = …
|hidden φ|target y
edge colour = sign of weight, thickness = |w|
fig.Press next layer to compute the network one layer at a time: the highlighted neurons are the ones being computed, with z (small) and a (large) inside each node. Drag x₁ and x₂ and every value downstream updates. Switch the hidden activation to ReLU and set x₂ = −1: h₂ gets z = −0.55 and outputs exactly 0. Flip the target y to see the same ŷ give a different loss.

Worked example: a 2-2-1 network

Two inputs, two sigmoid hidden units, one sigmoid output, cross-entropy loss. Parameters: W(1)=[0.4−0.20.30.6]W^{(1)}=\begin{bmatrix}0.4&-0.2\\0.3&0.6\end{bmatrix}, b(1)=(0.1,−0.1)b^{(1)}=(0.1,-0.1), W(2)=[0.7−0.5]W^{(2)}=\begin{bmatrix}0.7&-0.5\end{bmatrix}, b(2)=0.2b^{(2)}=0.2. Input x=(0.5, 1.0)x=(0.5,\,1.0), target y=1y=1.

QuantityComputationValue
z1(1)z^{(1)}_10.4⋅0.5+(−0.2)⋅1.0+0.10.4\cdot0.5+(-0.2)\cdot1.0+0.10.1000
z2(1)z^{(1)}_20.3⋅0.5+0.6⋅1.0−0.10.3\cdot0.5+0.6\cdot1.0-0.10.6500
a1(1)a^{(1)}_1σ(0.1)\sigma(0.1)0.5250
a2(1)a^{(1)}_2σ(0.65)\sigma(0.65)0.6570
z(2)z^{(2)}0.7⋅0.5250+(−0.5)⋅0.6570+0.20.7\cdot0.5250+(-0.5)\cdot0.6570+0.20.2390
y^=a(2)\hat y=a^{(2)}σ(0.2390)\sigma(0.2390)0.5595
L\mathcal{L}−log⁡0.5595-\log 0.5595 (since y=1y=1)0.5808
One forward pass (values rounded to 4 decimals; computed in float64)

The network gives the right class a probability of only 0.560.56, so the loss is well above zero. The backpropagation article continues from exactly these numbers.

Batching changes nothing about the arithmetic. Stack x=(0.5,1)x=(0.5,1) and a second input (1,−1)(1,-1) as rows of XX: XW(1)⊤+b(1)⊤XW^{(1)\top}+b^{(1)\top} gives the rows (0.1, 0.65)(0.1,\,0.65) and (0.7, −0.4)(0.7,\,-0.4), each one the per-example result.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3l1, l2 = nn.Linear(2, 2), nn.Linear(2, 1) # weights are (out, in)
4with torch.no_grad():
5 l1.weight.copy_(torch.tensor([[0.4, -0.2], [0.3, 0.6]]))
6 l1.bias.copy_(torch.tensor([0.1, -0.1]))
7 l2.weight.copy_(torch.tensor([[0.7, -0.5]]))
8 l2.bias.copy_(torch.tensor([0.2]))
9
10X = torch.tensor([[0.5, 1.0]]) # (B, 2), one row per example
11y = torch.tensor([[1.0]])
12a1 = torch.sigmoid(l1(X)) # X @ W1.T + b1, then sigma
13z2 = l2(a1)
14loss = F.binary_cross_entropy_with_logits(z2, y) # sigmoid + BCE, numerically stable
15print(a1, torch.sigmoid(z2), loss) # [[0.5250, 0.6570]] [[0.5595]] 0.5808
The same pass in PyTorch

Caching for backprop

The gradient of a weight needs the activation that fed into it (∂L/∂W(l)\partial\mathcal{L}/\partial W^{(l)} involves a(l−1)a^{(l-1)}), and the local slope of the activation needs z(l)z^{(l)}. So during training the forward pass stores every a(l)a^{(l)} (and z(l)z^{(l)}, or whatever the activation's derivative needs) instead of throwing them away.

Check yourself

A layer maps 784 features to 256 units and receives a batch of 64 examples, in the row-per-example convention. What are the shapes of W and Z?

Check yourself

In the worked example the network outputs ŷ = 0.5595. If the target were y = 0 instead of 1, what would the loss be?

Remember

  • ✓Recurrence: a(0)=xa^{(0)}=x, z(l)=W(l)a(l−1)+b(l)z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}, a(l)=φ(z(l))a^{(l)}=\varphi(z^{(l)}), then L=ℓ(a(L),y)\mathcal{L}=\ell(a^{(L)},y).
  • ✓W(l)W^{(l)} is nl×nl−1n_l\times n_{l-1} (out × in); check shapes at every layer.
  • ✓Batched: Z=AW⊤+bZ=AW^\top+b with one example per row; this is nn.Linear.
  • ✓Training caches each layer's activations for the backward pass; that cache dominates memory.
  • ✓Worked 2-2-1 example: ŷ = 0.5595, loss = 0.5808 for y = 1.
← back to Deep Learning roadmap