Definition
Weights and biases are a neuron's learnable numbers. Each weight multiplies one input, and its size and sign say how strongly and in which direction that input pushes the neuron. The bias is added after the weighted sum. It does not depend on any input, and it sets where the neuron switches on.
A neuron computes a pre-activation and passes it through a nonlinearity . Training adjusts nothing except the weights and biases, so everything a network knows is stored in them.
- the inputs to the neuron
- one weight per input
- the bias, a single number per neuron
- pre-activation (weighted sum plus bias)
- activation function and the neuron's output
One input: steepness and location
With one input and a sigmoid, factor out and the two parameters separate cleanly:
- the threshold, where and
- slope of the curve at the threshold, because
So sets how sharp the transition is and the sign of sets which way it faces. The bias, divided by the weight, sets where it happens. At the output is , which is set by the bias alone. A positive bias makes the neuron fire with no input at all.
Two inputs: a line and its normal
With two inputs, is a line. The weight vector is perpendicular to it and points to the side where . The bias moves the line along without turning it.
- normal vector of the boundary; its direction fixes the line's orientation
- length of w; larger means a sharper sigmoid across the line
- how far the line sits from the origin
Scaling by any leaves the line where it is but multiplies by . Direction and offset place the boundary. Overall scale sets how confident the neuron is near it.
Why the bias is not optional. Without , every boundary passes through the origin and the input always gives (sigmoid output , ReLU output ). A bias-free ReLU network is also positively homogeneous: for . It can never output a constant offset, and it assigns the same class as for every .
Worked example
| input x | w·x | z = w·x + b | a = σ(z) |
|---|---|---|---|
| (1, 3) | 2 − 3 = −1 | −0.5 | 0.3775 |
| (0, 0) | 0 | 0.5 | 0.6225 |
| (2, 1) | 4 − 1 = 3 | 3.5 | 0.9707 |
The zero input still gives : the neuron "fires" from its bias alone. In one dimension, , puts the threshold at with slope there. The outputs at and are and .
In a layer: a matrix and a vector
A layer of neurons stacks the weight vectors as the rows of a matrix and the biases into a vector. A layer from 784 inputs to 256 units has weights and biases, so parameters.
- weight from input to neuron (row is neuron 's )
- bias of neuron
- the bias is just the weight on an extra input fixed at 1
The augmented form shows that a bias is a weight whose input is always 1. Gradients treat it that way too: and , where . For a mini-batch, adds the same to every row by broadcasting, and the bias gradient is summed over the batch.
1import torch, torch.nn as nn23layer = nn.Linear(784, 256)4print(layer.weight.shape, layer.bias.shape) # (256, 784) (256,) - stored (out, in)5x = torch.randn(32, 784)6z = layer(x) # x @ W.T + b -> (32, 256)78nn.init.zeros_(layer.bias) # common choice; PyTorch's default is U(±1/sqrt(784))910# weight decay on weights only, not on biases (or norm parameters)11decay = [p for n, p in model.named_parameters() if p.ndim > 1]12no_decay = [p for n, p in model.named_parameters() if p.ndim <= 1]13opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.01},14 {"params": no_decay, "weight_decay": 0.0}], lr=3e-4)Initialising and regularising them
Weights must start random, otherwise every neuron in a layer computes the same thing and gets the same gradient (see weight initialization). Biases can start at 0, since random weights already break the symmetry. That is the usual choice in hand-written inits. PyTorch's nn.Linear default draws biases from instead. A useful exception is the output bias: set it to the log-odds of the class prior, , so an imbalanced classifier starts out predicting the base rate.
Biases are usually exempt from weight decay. An L2 penalty on limits how sensitive the output is to the input, which is the capacity you want to control. A bias does not change sensitivity, only position. Decaying it just pulls each threshold toward the origin. There are few biases, so they add little capacity anyway.
Check yourself
A sigmoid neuron has and . At which input is its output exactly ?
Check yourself
What is the shape of nn.Linear(100, 10).weight, and how many parameters does the layer have?
Remember
- ✓: weights scale inputs, the bias shifts the result.
- ✓1-D: = steepness, = location of the transition.
- ✓2-D: is the boundary's normal, and the line sits from the origin.
- ✓Without a bias every boundary passes through the origin.
- ✓A layer has and . The bias is the weight on a constant 1.
- ✓Biases usually start at 0 and are excluded from weight decay. Drop them right before BatchNorm.