Weights & biases

A weight scales how strongly an input drives a neuron; the bias shifts its threshold so the neuron can fire even when every input is zero.

7 min read691 wordscore0 viewsupdated 2026-10-09

Definition

Weights and biases are a neuron's learnable numbers. Each weight wiw_i multiplies one input, and its size and sign say how strongly and in which direction that input pushes the neuron. The bias bb is added after the weighted sum. It does not depend on any input, and it sets where the neuron switches on.

A neuron computes a pre-activation zz and passes it through a nonlinearity φ\varphi. Training adjusts nothing except the weights and biases, so everything a network knows is stored in them.

z=∑i=1nwixi+b=w⊤x+b,a=φ(z)z=\sum_{i=1}^{n} w_i x_i + b = w^\top x + b,\qquad a=\varphi(z)
x∈Rnx\in\mathbb{R}^n
the inputs to the neuron
w∈Rnw\in\mathbb{R}^n
one weight per input
b∈Rb\in\mathbb{R}
the bias, a single number per neuron
zz
pre-activation (weighted sum plus bias)
φ, a\varphi,\ a
activation function and the neuron's output

One input: steepness and location

With one input and a sigmoid, factor out ww and the two parameters separate cleanly:

σ(wx+b)=σ(w (x−x∗)),x∗=−bw,dadx∣x=x∗=w4\sigma(wx+b)=\sigma\bigl(w\,(x-x^\ast)\bigr),\qquad x^\ast=-\frac{b}{w},\qquad \left.\frac{d a}{d x}\right|_{x=x^\ast}=\frac{w}{4}
x∗x^\ast
the threshold, where z=0z=0 and a=0.5a=0.5
w/4w/4
slope of the curve at the threshold, because σ′(0)=1/4\sigma'(0)=1/4

So ∣w∣|w| sets how sharp the transition is and the sign of ww sets which way it faces. The bias, divided by the weight, sets where it happens. At x=0x=0 the output is σ(b)\sigma(b), which is set by the bias alone. A positive bias makes the neuron fire with no input at all.

Two inputs: a line and its normal

With two inputs, z=0z=0 is a line. The weight vector is perpendicular to it and points to the side where z>0z>0. The bias moves the line along ww without turning it.

{x:w⊤x+b=0},distance from origin=∣b∣∥w∥,signed distance of x=w⊤x+b∥w∥\{x : w^\top x+b=0\},\qquad \text{distance from origin}=\frac{|b|}{\lVert w\rVert},\qquad \text{signed distance of } x=\frac{w^\top x+b}{\lVert w\rVert}
ww
normal vector of the boundary; its direction fixes the line's orientation
∥w∥\lVert w\rVert
length of w; larger means a sharper sigmoid across the line
∣b∣/∥w∥|b|/\lVert w\rVert
how far the line sits from the origin

Scaling (w,b)(w,b) by any c>0c>0 leaves the line where it is but multiplies zz by cc. Direction and offset place the boundary. Overall scale sets how confident the neuron is near it.

one input: a = σ(w·x + b)00.51−6−3036xthreshold x = −b/w = 1.00slope there = w/4 = 0.38a(0) = σ(b) = 0.18two inputs: w₁x₁ + w₂x₂ + b = 0(0,0)0123→ w (normal) ┄ |b|/‖w‖ = 2.12errors 0/10 · best with b = 0 is 4/10grey dashed: σ(x) · red dot: output at x = 0
fig.Left: move w to sharpen or flip the sigmoid, and b to slide the threshold (orange). The red dot is the output at x = 0, which depends on b alone. Right: w₁ and w₂ turn the line and set the violet normal. b slides the line along the normal. Press "remove bias": the line is pinned to the origin and no rotation gets below 4 errors, because the two classes lie on the same rays from (0, 0).

Why the bias is not optional. Without bb, every boundary passes through the origin and the input x=0x=0 always gives z=0z=0 (sigmoid output 0.50.5, ReLU output 00). A bias-free ReLU network is also positively homogeneous: f(cx)=c f(x)f(cx)=c\,f(x) for c>0c>0. It can never output a constant offset, and it assigns cxcx the same class as xx for every c>0c>0.

Worked example

input xw·xz = w·x + ba = σ(z)
(1, 3)2 − 3 = −1−0.50.3775
(0, 0)00.50.6225
(2, 1)4 − 1 = 33.50.9707
One sigmoid neuron with w=(2,−1)w=(2,-1) and b=0.5b=0.5. The boundary is 2x1−x2+0.5=02x_1-x_2+0.5=0, at distance 0.5/5≈0.2240.5/\sqrt5\approx0.224 from the origin.

The zero input still gives a=0.62a=0.62: the neuron "fires" from its bias alone. In one dimension, w=2w=2, b=−3b=-3 puts the threshold at x∗=1.5x^\ast=1.5 with slope 0.50.5 there. The outputs at x=1x=1 and x=2x=2 are 0.2690.269 and 0.7310.731.

In a layer: a matrix and a vector

A layer of noutn_{out} neurons stacks the weight vectors as the rows of a matrix and the biases into a vector. A layer from 784 inputs to 256 units has 256×784=200,704256\times784=200{,}704 weights and 256256 biases, so 200,960200{,}960 parameters.

z=Wx+b=[Wb][x1],W∈Rnout×nin, b∈Rnoutz = Wx+b = \begin{bmatrix} W & b\end{bmatrix}\begin{bmatrix} x\\ 1\end{bmatrix},\qquad W\in\mathbb{R}^{n_{out}\times n_{in}},\ b\in\mathbb{R}^{n_{out}}
WjiW_{ji}
weight from input ii to neuron jj (row jj is neuron jj's ww)
bjb_j
bias of neuron jj
[ W b ][\,W\ b\,]
the bias is just the weight on an extra input fixed at 1

The augmented form shows that a bias is a weight whose input is always 1. Gradients treat it that way too: ∂L/∂Wji=δjxi\partial L/\partial W_{ji}=\delta_j x_i and ∂L/∂bj=δj⋅1\partial L/\partial b_j=\delta_j\cdot1, where δj=∂L/∂zj\delta_j=\partial L/\partial z_j. For a mini-batch, Z=XW⊤+bZ=XW^\top+b adds the same bb to every row by broadcasting, and the bias gradient is δ\delta summed over the batch.

python
1import torch, torch.nn as nn
2
3layer = nn.Linear(784, 256)
4print(layer.weight.shape, layer.bias.shape) # (256, 784) (256,) - stored (out, in)
5x = torch.randn(32, 784)
6z = layer(x) # x @ W.T + b -> (32, 256)
7
8nn.init.zeros_(layer.bias) # common choice; PyTorch's default is U(±1/sqrt(784))
9
10# weight decay on weights only, not on biases (or norm parameters)
11decay = [p for n, p in model.named_parameters() if p.ndim > 1]
12no_decay = [p for n, p in model.named_parameters() if p.ndim <= 1]
13opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.01},
14 {"params": no_decay, "weight_decay": 0.0}], lr=3e-4)

Initialising and regularising them

Weights must start random, otherwise every neuron in a layer computes the same thing and gets the same gradient (see weight initialization). Biases can start at 0, since random weights already break the symmetry. That is the usual choice in hand-written inits. PyTorch's nn.Linear default draws biases from U(−1/nin,1/nin)U(-1/\sqrt{n_{in}},1/\sqrt{n_{in}}) instead. A useful exception is the output bias: set it to the log-odds of the class prior, b=log⁡p1−pb=\log\frac{p}{1-p}, so an imbalanced classifier starts out predicting the base rate.

Biases are usually exempt from weight decay. An L2 penalty on ww limits how sensitive the output is to the input, which is the capacity you want to control. A bias does not change sensitivity, only position. Decaying it just pulls each threshold toward the origin. There are few biases, so they add little capacity anyway.

Check yourself

A sigmoid neuron has w=4w=4 and b=−2b=-2. At which input xx is its output exactly 0.50.5?

Check yourself

What is the shape of nn.Linear(100, 10).weight, and how many parameters does the layer have?

Remember

  • ✓z=w⊤x+bz=w^\top x+b: weights scale inputs, the bias shifts the result.
  • ✓1-D: ∣w∣|w| = steepness, x∗=−b/wx^\ast=-b/w = location of the transition.
  • ✓2-D: ww is the boundary's normal, and the line sits ∣b∣/∥w∥|b|/\lVert w\rVert from the origin.
  • ✓Without a bias every boundary passes through the origin.
  • ✓A layer has W∈Rnout×ninW\in\mathbb{R}^{n_{out}\times n_{in}} and b∈Rnoutb\in\mathbb{R}^{n_{out}}. The bias is the weight on a constant 1.
  • ✓Biases usually start at 0 and are excluded from weight decay. Drop them right before BatchNorm.
← back to Deep Learning roadmap