Definition
Weight initialization is the rule for drawing a network's starting weights. In a good scheme the weights are random, to break symmetry, and their variance is scaled to the layer's fan-in , so that signals neither shrink nor blow up as they pass through many layers.
Each layer multiplies its input by a weight matrix. If the weights are slightly too large, every layer amplifies the signal a little, and after 20 layers the activations overflow. If they are slightly too small, the signal fades to zero. Gradients flow back through the same matrices, so they explode or vanish the same way. Training then either diverges or never starts.
Why not zeros?
If every weight starts at 0 (or any shared constant), every hidden unit in a layer computes the same output and receives the same gradient. They stay identical forever, and a layer of 512 units behaves like one unit. With all-zero weights it is worse: the backpropagated error is zero, so the hidden layers get no gradient at all. Random weights break this symmetry. Biases, by contrast, can start at 0.
The variance argument
Take one pre-activation with . Assume the are i.i.d. with mean 0, independent of the inputs, and that the are i.i.d. with mean 0. Then the cross terms vanish, and each product has variance :
- fan-in: number of inputs summed into each unit
- variance of the initial weights (the thing we choose)
- variance of the layer's inputs (for non-zero-mean inputs, use )
To keep the same from layer to layer, choose . The backward pass gives the mirror condition , because gradients are summed over a unit's outgoing weights.
Xavier/Glorot (2010) targets tanh and sigmoid, which are roughly linear near 0. It compromises between the forward and backward conditions. He/Kaiming (2015) targets ReLU. ReLU zeroes the negative half of a symmetric , so , and the weights need twice the variance to make up for it:
- for tanh / sigmoid / linear layers; equals when
- for ReLU-family layers; keeps constant because
Normal or uniform only changes the shape, not the variance. has variance , so a target variance needs . That gives Xavier-uniform and He-uniform . The normal versions use standard deviation . Some libraries also truncate the normal at .
Simulation: 20 layers of width 512
| init | Var(w) | ReLU factor | ReLU z₁₀ | ReLU z₂₀ | tanh z₁₀ | tanh z₂₀ |
|---|---|---|---|---|---|---|
| N(0, 0.01²) | 1.0e−4 | 0.0256 | 1.4e−8 | 1.7e−16 | 3.3e−7 | 1.2e−13 |
| N(0, 1) | 1 | 256 | 1.4e12 | 1.7e24 | 22.2 (91% saturated) | 22.2 (91% saturated) |
| Xavier | 1/512 | 0.5 | 0.040 | 0.0013 | 0.24 | 0.17 |
| He | 2/512 | 1 | 1.29 | 1.37 | 0.78 | 0.79 |
| PyTorch default U(±1/√n) | 1/1536 | 1/6 | 1.7e−4 | 2.3e−8 | 0.0029 | 1.2e−5 |
Reading the table
- 1Layer 1 checks the formula. With , He gives , so . The simulation measured .
- 2ReLU + N(0, 1): variance per layer, so std . From layer 1 () to layer 5 that predicts ; the simulation measured .
- 3ReLU + Xavier: factor , so std per layer, and after 20 layers. Xavier is not wrong, it is meant for a different activation.
- 4tanh + Xavier decays slowly ( at layer 20), because shrinks values more than a linear function would. PyTorch's recommended tanh gain (std ) holds it at about in the same run.
- 5tanh + N(0, 1) looks stable (), but 91% of units have . There , so the gradients vanish.
What PyTorch actually does
nn.Linear and nn.Conv2d call kaiming_uniform_(weight, a=math.sqrt(5)) by default. The gain is , so the bound is . That is with variance , which is not He init: it is a third of Xavier's variance and a sixth of He's. Biases are drawn from the same range. The default is fine for shallow nets and for nets with normalization layers or residual connections. For a deep plain ReLU stack, set the init explicitly.
1import torch.nn as nn23def init_weights(m):4 if isinstance(m, (nn.Linear, nn.Conv2d)):5 nn.init.kaiming_normal_(m.weight, nonlinearity="relu") # std = sqrt(2 / fan_in)6 # tanh net instead: nn.init.xavier_uniform_(m.weight, gain=nn.init.calculate_gain("tanh"))7 if m.bias is not None:8 nn.init.zeros_(m.bias)910model = nn.Sequential(*[layer for _ in range(20) for layer in (nn.Linear(512, 512), nn.ReLU())])11model.apply(init_weights)Check yourself
A ReLU layer has fan-in 256. Which standard deviation does He-normal initialization use?
Check yourself
A deep ReLU network is initialised with Xavier (n_in = n_out). Going forward, what happens to the activation variance at each layer?
Remember
- ✓Random init breaks symmetry. Zero or constant weights keep all units identical.
- ✓ for independent, zero-mean weights and inputs.
- ✓Xavier: for tanh/sigmoid. He: for ReLU.
- ✓Uniform has variance , so pick .
- ✓PyTorch's
nn.Lineardefault is , variance , which is not He. - ✓The wrong scale compounds geometrically with depth: ×16 or ×0.16 in std per layer means overflow or underflow within 10–20 layers.