# Weight initialization

> Scale initial weights by fan-in (Xavier, He) so activation variance stays constant across layers.

Source: https://www.mlfoundry.in/learn/weight-initialization  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 8 min read · updated 2026-10-09

**Weight initialization:** is the rule for drawing a network's starting weights. In a good scheme the weights are random, to break symmetry, and their variance is scaled to the layer's **fan-in** $n_{in}$, so that signals neither shrink nor blow up as they pass through many layers.

Each layer multiplies its input by a weight matrix. If the weights are slightly too large, every layer amplifies the signal a little, and after 20 layers the activations overflow. If they are slightly too small, the signal fades to zero. Gradients flow back through the same matrices, so they explode or vanish the same way. Training then either diverges or never starts.

## Why not zeros?

If every weight starts at 0 (or any shared constant), every hidden unit in a layer computes the same output and receives the same gradient. They stay identical forever, and a layer of 512 units behaves like one unit. With all-zero weights it is worse: the backpropagated error $\delta = W^\top(\cdot)$ is zero, so the hidden layers get no gradient at all. Random weights break this **symmetry**. Biases, by contrast, can start at 0.

## The variance argument

Take one pre-activation $z=\sum_{i=1}^{n} w_i x_i$ with $n=n_{in}$. Assume the $w_i$ are i.i.d. with mean 0, independent of the inputs, and that the $x_i$ are i.i.d. with mean 0. Then the cross terms vanish, and each product has variance $\operatorname{Var}(w)\operatorname{Var}(x)$:

$$
\operatorname{Var}(z)=n_{in}\operatorname{Var}(w)\operatorname{Var}(x)
$$

- $n_{in}$: fan-in: number of inputs summed into each unit
- $\operatorname{Var}(w)$: variance of the initial weights (the thing we choose)
- $\operatorname{Var}(x)$: variance of the layer's inputs (for non-zero-mean inputs, use $\mathbb{E}[x^2]$)

To keep $\operatorname{Var}(z)$ the same from layer to layer, choose $n_{in}\operatorname{Var}(w)\approx1$. The backward pass gives the mirror condition $n_{out}\operatorname{Var}(w)\approx1$, because gradients are summed over a unit's $n_{out}$ outgoing weights.

**Xavier/Glorot (2010)** targets tanh and sigmoid, which are roughly linear near 0. It compromises between the forward and backward conditions. **He/Kaiming (2015)** targets ReLU. ReLU zeroes the negative half of a symmetric $z$, so $\mathbb{E}[x^2]=\tfrac12\operatorname{Var}(z_{prev})$, and the weights need twice the variance to make up for it:

$$
\text{Xavier: }\operatorname{Var}(w)=\frac{2}{n_{in}+n_{out}}\qquad\qquad \text{He: }\operatorname{Var}(w)=\frac{2}{n_{in}}
$$

- $Xavier$: for tanh / sigmoid / linear layers; equals $1/n$ when $n_{in}=n_{out}=n$
- $He$: for ReLU-family layers; keeps $\operatorname{Var}(z)$ constant because $n_{in}\cdot\frac{2}{n_{in}}\cdot\frac12=1$

**Normal or uniform** only changes the shape, not the variance. $U(-a,a)$ has variance $a^2/3$, so a target variance $v$ needs $a=\sqrt{3v}$. That gives Xavier-uniform $a=\sqrt{6/(n_{in}+n_{out})}$ and He-uniform $a=\sqrt{6/n_{in}}$. The normal versions use standard deviation $\sqrt v$. Some libraries also truncate the normal at $\pm2\sigma$.

> Figure: A seeded forward pass through up to 20 bias-free layers of 80 units, with the same random draws for every scheme. Pick ReLU and switch between He (flat at about 1), Xavier (fades by about ×0.71 per layer) and N(0, 1) (explodes). Then pick tanh with N(0, 1): the std of z stays large, but the histogram piles up at ±1 because the units are saturated, so gradients vanish anyway. The PyTorch default collapses in both cases.

## Simulation: 20 layers of width 512

numpy, 1000 inputs $x\sim\mathcal N(0,I)$, 20 square layers of width 512, no biases, seed 0. The cells give $\mathrm{std}(z)$ at layers 10 and 20. "Factor" is the analytic per-layer variance multiplier under ReLU, $n\operatorname{Var}(w)/2$.

| init | Var(w) | ReLU factor | ReLU z₁₀ | ReLU z₂₀ | tanh z₁₀ | tanh z₂₀ |
| --- | --- | --- | --- | --- | --- | --- |
| N(0, 0.01²) | 1.0e−4 | 0.0256 | 1.4e−8 | 1.7e−16 | 3.3e−7 | 1.2e−13 |
| N(0, 1) | 1 | 256 | 1.4e12 | 1.7e24 | 22.2 (91% saturated) | 22.2 (91% saturated) |
| Xavier | 1/512 | 0.5 | 0.040 | 0.0013 | 0.24 | 0.17 |
| He | 2/512 | 1 | 1.29 | 1.37 | 0.78 | 0.79 |
| PyTorch default U(±1/√n) | 1/1536 | 1/6 | 1.7e−4 | 2.3e−8 | 0.0029 | 1.2e−5 |

**Reading the table**

1. **Layer 1** checks the formula. With $\operatorname{Var}(x)=1$, He gives $\operatorname{Var}(z)=512\cdot\frac{2}{512}\cdot1=2$, so $\mathrm{std}=1.414$. The simulation measured $1.42$.
2. **ReLU + N(0, 1)**: variance $\times256$ per layer, so std $\times16$. From layer 1 ($22.7$) to layer 5 that predicts $22.7\cdot16^4\approx1.5\times10^6$; the simulation measured $1.4\times10^6$.
3. **ReLU + Xavier**: factor $0.5$, so std $\times0.707$ per layer, and $0.707^{19}\approx1.4\times10^{-3}$ after 20 layers. Xavier is not wrong, it is meant for a different activation.
4. **tanh + Xavier** decays slowly ($0.17$ at layer 20), because $\tanh$ shrinks values more than a linear function would. PyTorch's recommended tanh gain $5/3$ (std $\frac53\sqrt{1/n}$) holds it at about $1.08$ in the same run.
5. **tanh + N(0, 1)** looks stable ($\mathrm{std}(z)\approx22$), but 91% of units have $|h|>0.99$. There $\tanh'\approx0$, so the gradients vanish.

## What PyTorch actually does

`nn.Linear` and `nn.Conv2d` call `kaiming_uniform_(weight, a=math.sqrt(5))` by default. The gain is $\sqrt{2/(1+a^2)}=\sqrt{1/3}$, so the bound is $\sqrt{1/3}\cdot\sqrt{3/n_{in}}=1/\sqrt{n_{in}}$. That is $U(-1/\sqrt{n_{in}},1/\sqrt{n_{in}})$ with variance $1/(3n_{in})$, which is **not** He init: it is a third of Xavier's variance and a sixth of He's. Biases are drawn from the same range. The default is fine for shallow nets and for nets with normalization layers or residual connections. For a deep plain ReLU stack, set the init explicitly.

```python
import torch.nn as nn

def init_weights(m):
    if isinstance(m, (nn.Linear, nn.Conv2d)):
        nn.init.kaiming_normal_(m.weight, nonlinearity="relu")   # std = sqrt(2 / fan_in)
        # tanh net instead: nn.init.xavier_uniform_(m.weight, gain=nn.init.calculate_gain("tanh"))
        if m.bias is not None:
            nn.init.zeros_(m.bias)

model = nn.Sequential(*[layer for _ in range(20) for layer in (nn.Linear(512, 512), nn.ReLU())])
model.apply(init_weights)
```

> **Variance is only half the story:** The argument controls the variance at step 0, for one particular activation. Change the activation (GELU, SiLU, leaky ReLU), the fan mode (`fan_in` preserves forward signals, `fan_out` preserves backward gradients) or add residual branches, and the right scale changes. Residual nets often scale or zero-initialise the last layer of each branch, so that every block starts close to the identity.

> **Why modern nets are forgiving:** BatchNorm and LayerNorm re-standardise activations at every layer, and residual connections give gradients a path around each block. Both remove much of the sensitivity to the initial scale. Initialization matters most in deep **plain** stacks, the case this article's simulation shows.

**Question:** A ReLU layer has fan-in 256. Which standard deviation does He-normal initialization use?

- A. 1/256
- B. 1/16
- C. √2/16 ≈ 0.088
- D. 2/256

**Answer:** C. √2/16 ≈ 0.088 — He sets $\operatorname{Var}(w)=2/n_{in}=2/256$, so $\mathrm{std}=\sqrt{2/256}=\sqrt2/16\approx0.088$. The option $2/256$ is the variance, not the std.

**Question:** A deep ReLU network is initialised with Xavier (n_in = n_out). Going forward, what happens to the activation variance at each layer?

- A. It stays constant
- B. It halves
- C. It doubles
- D. It drops to zero after one layer

**Answer:** B. It halves — Xavier gives $n\operatorname{Var}(w)=1$, and ReLU keeps only half of $\mathbb{E}[z^2]$, so $\operatorname{Var}(z)$ is multiplied by $1\cdot\frac12$ per layer. After 20 layers the std is down to about $0.707^{19}\approx10^{-3}$. He doubles $\operatorname{Var}(w)$ to cancel the $\frac12$.

**Key points**

- Random init breaks symmetry. Zero or constant weights keep all units identical.
- $\operatorname{Var}(z)=n_{in}\operatorname{Var}(w)\operatorname{Var}(x)$ for independent, zero-mean weights and inputs.
- Xavier: $\operatorname{Var}(w)=2/(n_{in}+n_{out})$ for tanh/sigmoid. He: $2/n_{in}$ for ReLU.
- Uniform $U(-a,a)$ has variance $a^2/3$, so pick $a=\sqrt{3\,\operatorname{Var}}$.
- PyTorch's `nn.Linear` default is $U(\pm1/\sqrt{n_{in}})$, variance $1/(3n_{in})$, which is not He.
- The wrong scale compounds geometrically with depth: ×16 or ×0.16 in std per layer means overflow or underflow within 10–20 layers.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
