Weight initialization

Scale initial weights by fan-in (Xavier, He) so activation variance stays constant across layers.

8 min read703 wordscore0 viewsupdated 2026-10-09

Definition

Weight initialization is the rule for drawing a network's starting weights. In a good scheme the weights are random, to break symmetry, and their variance is scaled to the layer's fan-in ninn_{in}, so that signals neither shrink nor blow up as they pass through many layers.

Each layer multiplies its input by a weight matrix. If the weights are slightly too large, every layer amplifies the signal a little, and after 20 layers the activations overflow. If they are slightly too small, the signal fades to zero. Gradients flow back through the same matrices, so they explode or vanish the same way. Training then either diverges or never starts.

Why not zeros?

If every weight starts at 0 (or any shared constant), every hidden unit in a layer computes the same output and receives the same gradient. They stay identical forever, and a layer of 512 units behaves like one unit. With all-zero weights it is worse: the backpropagated error δ=W⊤(⋅)\delta = W^\top(\cdot) is zero, so the hidden layers get no gradient at all. Random weights break this symmetry. Biases, by contrast, can start at 0.

The variance argument

Take one pre-activation z=∑i=1nwixiz=\sum_{i=1}^{n} w_i x_i with n=ninn=n_{in}. Assume the wiw_i are i.i.d. with mean 0, independent of the inputs, and that the xix_i are i.i.d. with mean 0. Then the cross terms vanish, and each product has variance Var⁡(w)Var⁡(x)\operatorname{Var}(w)\operatorname{Var}(x):

Var⁡(z)=ninVar⁡(w)Var⁡(x)\operatorname{Var}(z)=n_{in}\operatorname{Var}(w)\operatorname{Var}(x)
ninn_{in}
fan-in: number of inputs summed into each unit
Var⁡(w)\operatorname{Var}(w)
variance of the initial weights (the thing we choose)
Var⁡(x)\operatorname{Var}(x)
variance of the layer's inputs (for non-zero-mean inputs, use E[x2]\mathbb{E}[x^2])

To keep Var⁡(z)\operatorname{Var}(z) the same from layer to layer, choose ninVar⁡(w)≈1n_{in}\operatorname{Var}(w)\approx1. The backward pass gives the mirror condition noutVar⁡(w)≈1n_{out}\operatorname{Var}(w)\approx1, because gradients are summed over a unit's noutn_{out} outgoing weights.

Xavier/Glorot (2010) targets tanh and sigmoid, which are roughly linear near 0. It compromises between the forward and backward conditions. He/Kaiming (2015) targets ReLU. ReLU zeroes the negative half of a symmetric zz, so E[x2]=12Var⁡(zprev)\mathbb{E}[x^2]=\tfrac12\operatorname{Var}(z_{prev}), and the weights need twice the variance to make up for it:

Xavier: Var⁡(w)=2nin+noutHe: Var⁡(w)=2nin\text{Xavier: }\operatorname{Var}(w)=\frac{2}{n_{in}+n_{out}}\qquad\qquad \text{He: }\operatorname{Var}(w)=\frac{2}{n_{in}}
XavierXavier
for tanh / sigmoid / linear layers; equals 1/n1/n when nin=nout=nn_{in}=n_{out}=n
HeHe
for ReLU-family layers; keeps Var⁡(z)\operatorname{Var}(z) constant because nin⋅2nin⋅12=1n_{in}\cdot\frac{2}{n_{in}}\cdot\frac12=1

Normal or uniform only changes the shape, not the variance. U(−a,a)U(-a,a) has variance a2/3a^2/3, so a target variance vv needs a=3va=\sqrt{3v}. That gives Xavier-uniform a=6/(nin+nout)a=\sqrt{6/(n_{in}+n_{out})} and He-uniform a=6/nina=\sqrt{6/n_{in}}. The normal versions use standard deviation v\sqrt v. Some libraries also truncate the normal at ±2σ\pm2\sigma.

std of pre-activations z, per layer (log scale)1e-61e-41e-211e21e41e6151012layervalues beyond 1e±6 are pinned to the edgeactivations h at layer 12024+std(h) = 0.825h = 0: 56% · h ≥ 4: 1%● N(0, 0.01²)● N(0, 1)● Xavier● He● PyTorch default
activation
init
rule Var = 2/n_inn·Var(w) = 2.00variance ×1.00 per layerstd(z) at layer 12 = 1.36
80 units per layer, 100 inputs, no biases. Every scheme uses the same random draws, only scaled differently.
fig.A seeded forward pass through up to 20 bias-free layers of 80 units, with the same random draws for every scheme. Pick ReLU and switch between He (flat at about 1), Xavier (fades by about ×0.71 per layer) and N(0, 1) (explodes). Then pick tanh with N(0, 1): the std of z stays large, but the histogram piles up at ±1 because the units are saturated, so gradients vanish anyway. The PyTorch default collapses in both cases.

Simulation: 20 layers of width 512

initVar(w)ReLU factorReLU z₁₀ReLU z₂₀tanh z₁₀tanh z₂₀
N(0, 0.01²)1.0e−40.02561.4e−81.7e−163.3e−71.2e−13
N(0, 1)12561.4e121.7e2422.2 (91% saturated)22.2 (91% saturated)
Xavier1/5120.50.0400.00130.240.17
He2/51211.291.370.780.79
PyTorch default U(±1/√n)1/15361/61.7e−42.3e−80.00291.2e−5
numpy, 1000 inputs x∼N(0,I)x\sim\mathcal N(0,I), 20 square layers of width 512, no biases, seed 0. The cells give std(z)\mathrm{std}(z) at layers 10 and 20. "Factor" is the analytic per-layer variance multiplier under ReLU, nVar⁡(w)/2n\operatorname{Var}(w)/2.

Reading the table

  1. 1Layer 1 checks the formula. With Var⁡(x)=1\operatorname{Var}(x)=1, He gives Var⁡(z)=512⋅2512⋅1=2\operatorname{Var}(z)=512\cdot\frac{2}{512}\cdot1=2, so std=1.414\mathrm{std}=1.414. The simulation measured 1.421.42.
  2. 2ReLU + N(0, 1): variance ×256\times256 per layer, so std ×16\times16. From layer 1 (22.722.7) to layer 5 that predicts 22.7⋅164≈1.5×10622.7\cdot16^4\approx1.5\times10^6; the simulation measured 1.4×1061.4\times10^6.
  3. 3ReLU + Xavier: factor 0.50.5, so std ×0.707\times0.707 per layer, and 0.70719≈1.4×10−30.707^{19}\approx1.4\times10^{-3} after 20 layers. Xavier is not wrong, it is meant for a different activation.
  4. 4tanh + Xavier decays slowly (0.170.17 at layer 20), because tanh⁡\tanh shrinks values more than a linear function would. PyTorch's recommended tanh gain 5/35/3 (std 531/n\frac53\sqrt{1/n}) holds it at about 1.081.08 in the same run.
  5. 5tanh + N(0, 1) looks stable (std(z)≈22\mathrm{std}(z)\approx22), but 91% of units have ∣h∣>0.99|h|>0.99. There tanh⁡′≈0\tanh'\approx0, so the gradients vanish.

What PyTorch actually does

nn.Linear and nn.Conv2d call kaiming_uniform_(weight, a=math.sqrt(5)) by default. The gain is 2/(1+a2)=1/3\sqrt{2/(1+a^2)}=\sqrt{1/3}, so the bound is 1/3⋅3/nin=1/nin\sqrt{1/3}\cdot\sqrt{3/n_{in}}=1/\sqrt{n_{in}}. That is U(−1/nin,1/nin)U(-1/\sqrt{n_{in}},1/\sqrt{n_{in}}) with variance 1/(3nin)1/(3n_{in}), which is not He init: it is a third of Xavier's variance and a sixth of He's. Biases are drawn from the same range. The default is fine for shallow nets and for nets with normalization layers or residual connections. For a deep plain ReLU stack, set the init explicitly.

python
1import torch.nn as nn
2
3def init_weights(m):
4 if isinstance(m, (nn.Linear, nn.Conv2d)):
5 nn.init.kaiming_normal_(m.weight, nonlinearity="relu") # std = sqrt(2 / fan_in)
6 # tanh net instead: nn.init.xavier_uniform_(m.weight, gain=nn.init.calculate_gain("tanh"))
7 if m.bias is not None:
8 nn.init.zeros_(m.bias)
9
10model = nn.Sequential(*[layer for _ in range(20) for layer in (nn.Linear(512, 512), nn.ReLU())])
11model.apply(init_weights)

Check yourself

A ReLU layer has fan-in 256. Which standard deviation does He-normal initialization use?

Check yourself

A deep ReLU network is initialised with Xavier (n_in = n_out). Going forward, what happens to the activation variance at each layer?

Remember

  • ✓Random init breaks symmetry. Zero or constant weights keep all units identical.
  • ✓Var⁡(z)=ninVar⁡(w)Var⁡(x)\operatorname{Var}(z)=n_{in}\operatorname{Var}(w)\operatorname{Var}(x) for independent, zero-mean weights and inputs.
  • ✓Xavier: Var⁡(w)=2/(nin+nout)\operatorname{Var}(w)=2/(n_{in}+n_{out}) for tanh/sigmoid. He: 2/nin2/n_{in} for ReLU.
  • ✓Uniform U(−a,a)U(-a,a) has variance a2/3a^2/3, so pick a=3 Var⁡a=\sqrt{3\,\operatorname{Var}}.
  • ✓PyTorch's nn.Linear default is U(±1/nin)U(\pm1/\sqrt{n_{in}}), variance 1/(3nin)1/(3n_{in}), which is not He.
  • ✓The wrong scale compounds geometrically with depth: ×16 or ×0.16 in std per layer means overflow or underflow within 10–20 layers.
← back to Deep Learning roadmap