# Forward propagation

> Compute each layer from the previous one, z = Wa + b then a = φ(z), until the output and the loss come out the end.

Source: https://www.mlfoundry.in/learn/forward-propagation  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 6 min read · updated 2026-10-09

**Forward propagation:** is the evaluation of a network on an input: starting from $a^{(0)}=x$, each layer applies an affine map followed by a nonlinearity to the previous layer's output, and the last layer's output is compared with the target to give a scalar loss.

A feed-forward network is a composition of functions, $f(x)=f^{(L)}\big(\cdots f^{(2)}(f^{(1)}(x))\big)$. Forward propagation evaluates that composition from the inside out, one layer at a time. Nothing is learned during this pass; the weights are fixed.

It is used in two places. At **inference** it is the whole job: the output is the prediction. During **training** it is the first half of every step, and it must leave behind the intermediate values that the backward pass needs.

## The layer recurrence

$$
a^{(0)}=x,\qquad z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=\varphi^{(l)}\!\big(z^{(l)}\big),\qquad l=1,\dots,L
$$

- $a^{(l)}\in\mathbb{R}^{n_l}$: activations (outputs) of layer $l$; $n_l$ is its width
- $W^{(l)}\in\mathbb{R}^{n_l\times n_{l-1}}$: weight matrix; row $j$ holds the incoming weights of neuron $j$
- $b^{(l)}\in\mathbb{R}^{n_l}$: bias vector, one entry per neuron
- $z^{(l)}$: pre-activation: the weighted sum before the nonlinearity
- $\varphi^{(l)}$: activation function, applied elementwise (ReLU, sigmoid, …)

The output $\hat y=a^{(L)}$ then goes into the loss, $\mathcal{L}=\ell(\hat y, y)$. For a binary classifier with a sigmoid output, that is the cross-entropy $\ell=-[y\log\hat y+(1-y)\log(1-\hat y)]$.

**Shapes are the best bug detector.** $W^{(l)}$ maps $n_{l-1}$ numbers to $n_l$ numbers, so it is $n_l\times n_{l-1}$ (out × in). A layer from 784 inputs to 256 units has a $256\times784$ weight matrix and 256 biases: $200{,}960$ parameters.

## Batched form

In practice a whole mini-batch goes through at once. Stack the $B$ examples as **rows** of a matrix, and each layer becomes one matrix multiply:

$$
Z^{(l)}=A^{(l-1)}\,W^{(l)\top}+\mathbf{1}\,b^{(l)\top},\qquad A^{(l)}=\varphi\big(Z^{(l)}\big)
$$

- $A^{(l-1)}\in\mathbb{R}^{B\times n_{l-1}}$: previous layer's activations, one row per example
- $Z^{(l)}\in\mathbb{R}^{B\times n_l}$: pre-activations for the whole batch
- $\mathbf{1}\,b^{(l)\top}$: the bias added to every row (broadcasting)

This is exactly what PyTorch's `nn.Linear` computes: it stores `weight` with shape `(out, in)` and returns `x @ weight.T + bias`. Each layer costs $B\,n_l\,n_{l-1}$ multiply-adds, which is why the forward pass is dominated by matrix multiplication and runs so well on GPUs.

> Figure: Press next layer to compute the network one layer at a time: the highlighted neurons are the ones being computed, with z (small) and a (large) inside each node. Drag x₁ and x₂ and every value downstream updates. Switch the hidden activation to ReLU and set x₂ = −1: h₂ gets z = −0.55 and outputs exactly 0. Flip the target y to see the same ŷ give a different loss.

## Worked example: a 2-2-1 network

Two inputs, two sigmoid hidden units, one sigmoid output, cross-entropy loss. Parameters: $W^{(1)}=\begin{bmatrix}0.4&-0.2\\0.3&0.6\end{bmatrix}$, $b^{(1)}=(0.1,-0.1)$, $W^{(2)}=\begin{bmatrix}0.7&-0.5\end{bmatrix}$, $b^{(2)}=0.2$. Input $x=(0.5,\,1.0)$, target $y=1$.

One forward pass (values rounded to 4 decimals; computed in float64)

| Quantity | Computation | Value |
| --- | --- | --- |
| $z^{(1)}_1$ | $0.4\cdot0.5+(-0.2)\cdot1.0+0.1$ | 0.1000 |
| $z^{(1)}_2$ | $0.3\cdot0.5+0.6\cdot1.0-0.1$ | 0.6500 |
| $a^{(1)}_1$ | $\sigma(0.1)$ | 0.5250 |
| $a^{(1)}_2$ | $\sigma(0.65)$ | 0.6570 |
| $z^{(2)}$ | $0.7\cdot0.5250+(-0.5)\cdot0.6570+0.2$ | 0.2390 |
| $\hat y=a^{(2)}$ | $\sigma(0.2390)$ | 0.5595 |
| $\mathcal{L}$ | $-\log 0.5595$ (since $y=1$) | 0.5808 |

The network gives the right class a probability of only $0.56$, so the loss is well above zero. The backpropagation article continues from exactly these numbers.

Batching changes nothing about the arithmetic. Stack $x=(0.5,1)$ and a second input $(1,-1)$ as rows of $X$: $XW^{(1)\top}+b^{(1)\top}$ gives the rows $(0.1,\,0.65)$ and $(0.7,\,-0.4)$, each one the per-example result.

```python
import torch, torch.nn as nn, torch.nn.functional as F

l1, l2 = nn.Linear(2, 2), nn.Linear(2, 1)        # weights are (out, in)
with torch.no_grad():
    l1.weight.copy_(torch.tensor([[0.4, -0.2], [0.3, 0.6]]))
    l1.bias.copy_(torch.tensor([0.1, -0.1]))
    l2.weight.copy_(torch.tensor([[0.7, -0.5]]))
    l2.bias.copy_(torch.tensor([0.2]))

X = torch.tensor([[0.5, 1.0]])                   # (B, 2), one row per example
y = torch.tensor([[1.0]])
a1 = torch.sigmoid(l1(X))                        # X @ W1.T + b1, then sigma
z2 = l2(a1)
loss = F.binary_cross_entropy_with_logits(z2, y) # sigmoid + BCE, numerically stable
print(a1, torch.sigmoid(z2), loss)               # [[0.5250, 0.6570]]  [[0.5595]]  0.5808
```

The same pass in PyTorch

## Caching for backprop

The gradient of a weight needs the activation that fed into it ($\partial\mathcal{L}/\partial W^{(l)}$ involves $a^{(l-1)}$), and the local slope of the activation needs $z^{(l)}$. So during training the forward pass **stores** every $a^{(l)}$ (and $z^{(l)}$, or whatever the activation's derivative needs) instead of throwing them away.

> **Activations, not weights, fill GPU memory:** The cache holds $B\times n_l$ numbers for every layer, so memory grows with batch size × depth × width. That is usually what runs out first in training. At inference, wrap the pass in `torch.no_grad()` (or `torch.inference_mode()`) so nothing is cached; gradient checkpointing trades the other way, recomputing activations during the backward pass to save memory.

> **Why the nonlinearity matters here:** Remove $\varphi$ and the recurrence collapses: $W^{(2)}(W^{(1)}x+b^{(1)})+b^{(2)}$ is a single affine map, however many layers you stack. The forward pass is only more expressive than linear regression because of the elementwise bend between the matrix multiplies.

**Question:** A layer maps 784 features to 256 units and receives a batch of 64 examples, in the row-per-example convention. What are the shapes of W and Z?

- A. W: 784×256, Z: 256×64
- B. W: 256×784, Z: 64×256
- C. W: 256×784, Z: 64×784
- D. W: 64×256, Z: 256×784

**Answer:** B. W: 256×784, Z: 64×256 — $W$ is out × in, so $256\times784$. Then $Z=AW^\top+b$ with $A$ of shape $64\times784$ gives $(64\times784)(784\times256)=64\times256$: one row of 256 pre-activations per example.

**Question:** In the worked example the network outputs ŷ = 0.5595. If the target were y = 0 instead of 1, what would the loss be?

- A. 0.5808
- B. 0.4405
- C. 0.8198
- D. 0.2390

**Answer:** C. 0.8198 — With $y=0$ the cross-entropy is $-\log(1-\hat y)=-\log 0.4405\approx0.8198$. The forward pass up to $\hat y$ is identical; only the last step depends on the target.

**Key points**

- Recurrence: $a^{(0)}=x$, $z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$, $a^{(l)}=\varphi(z^{(l)})$, then $\mathcal{L}=\ell(a^{(L)},y)$.
- $W^{(l)}$ is $n_l\times n_{l-1}$ (out × in); check shapes at every layer.
- Batched: $Z=AW^\top+b$ with one example per row; this is `nn.Linear`.
- Training caches each layer's activations for the backward pass; that cache dominates memory.
- Worked 2-2-1 example: ŷ = 0.5595, loss = 0.5808 for y = 1.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
