Definition
Forward propagation is the evaluation of a network on an input: starting from , each layer applies an affine map followed by a nonlinearity to the previous layer's output, and the last layer's output is compared with the target to give a scalar loss.
A feed-forward network is a composition of functions, . Forward propagation evaluates that composition from the inside out, one layer at a time. Nothing is learned during this pass; the weights are fixed.
It is used in two places. At inference it is the whole job: the output is the prediction. During training it is the first half of every step, and it must leave behind the intermediate values that the backward pass needs.
The layer recurrence
- activations (outputs) of layer ; is its width
- weight matrix; row holds the incoming weights of neuron
- bias vector, one entry per neuron
- pre-activation: the weighted sum before the nonlinearity
- activation function, applied elementwise (ReLU, sigmoid, …)
The output then goes into the loss, . For a binary classifier with a sigmoid output, that is the cross-entropy .
Shapes are the best bug detector. maps numbers to numbers, so it is (out × in). A layer from 784 inputs to 256 units has a weight matrix and 256 biases: parameters.
Batched form
In practice a whole mini-batch goes through at once. Stack the examples as rows of a matrix, and each layer becomes one matrix multiply:
- previous layer's activations, one row per example
- pre-activations for the whole batch
- the bias added to every row (broadcasting)
This is exactly what PyTorch's nn.Linear computes: it stores weight with shape (out, in) and returns x @ weight.T + bias. Each layer costs multiply-adds, which is why the forward pass is dominated by matrix multiplication and runs so well on GPUs.
Worked example: a 2-2-1 network
Two inputs, two sigmoid hidden units, one sigmoid output, cross-entropy loss. Parameters: , , , . Input , target .
| Quantity | Computation | Value |
|---|---|---|
| 0.1000 | ||
| 0.6500 | ||
| 0.5250 | ||
| 0.6570 | ||
| 0.2390 | ||
| 0.5595 | ||
| (since ) | 0.5808 |
The network gives the right class a probability of only , so the loss is well above zero. The backpropagation article continues from exactly these numbers.
Batching changes nothing about the arithmetic. Stack and a second input as rows of : gives the rows and , each one the per-example result.
1import torch, torch.nn as nn, torch.nn.functional as F23l1, l2 = nn.Linear(2, 2), nn.Linear(2, 1) # weights are (out, in)4with torch.no_grad():5 l1.weight.copy_(torch.tensor([[0.4, -0.2], [0.3, 0.6]]))6 l1.bias.copy_(torch.tensor([0.1, -0.1]))7 l2.weight.copy_(torch.tensor([[0.7, -0.5]]))8 l2.bias.copy_(torch.tensor([0.2]))910X = torch.tensor([[0.5, 1.0]]) # (B, 2), one row per example11y = torch.tensor([[1.0]])12a1 = torch.sigmoid(l1(X)) # X @ W1.T + b1, then sigma13z2 = l2(a1)14loss = F.binary_cross_entropy_with_logits(z2, y) # sigmoid + BCE, numerically stable15print(a1, torch.sigmoid(z2), loss) # [[0.5250, 0.6570]] [[0.5595]] 0.5808Caching for backprop
The gradient of a weight needs the activation that fed into it ( involves ), and the local slope of the activation needs . So during training the forward pass stores every (and , or whatever the activation's derivative needs) instead of throwing them away.
Check yourself
A layer maps 784 features to 256 units and receives a batch of 64 examples, in the row-per-example convention. What are the shapes of W and Z?
Check yourself
In the worked example the network outputs ŷ = 0.5595. If the target were y = 0 instead of 1, what would the loss be?
Remember
- ✓Recurrence: , , , then .
- ✓ is (out × in); check shapes at every layer.
- ✓Batched: with one example per row; this is
nn.Linear. - ✓Training caches each layer's activations for the backward pass; that cache dominates memory.
- ✓Worked 2-2-1 example: ŷ = 0.5595, loss = 0.5808 for y = 1.