# Weights & biases

> A weight scales how strongly an input drives a neuron; the bias shifts its threshold so the neuron can fire even when every input is zero.

Source: https://www.mlfoundry.in/learn/weights-biases  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 7 min read · updated 2026-10-09

**Weights and biases:** are a neuron's learnable numbers. Each **weight** $w_i$ multiplies one input, and its size and sign say how strongly and in which direction that input pushes the neuron. The **bias** $b$ is added after the weighted sum. It does not depend on any input, and it sets where the neuron switches on.

A neuron computes a pre-activation $z$ and passes it through a nonlinearity $\varphi$. Training adjusts nothing except the weights and biases, so everything a network knows is stored in them.

$$
z=\sum_{i=1}^{n} w_i x_i + b = w^\top x + b,\qquad a=\varphi(z)
$$

- $x\in\mathbb{R}^n$: the inputs to the neuron
- $w\in\mathbb{R}^n$: one weight per input
- $b\in\mathbb{R}$: the bias, a single number per neuron
- $z$: pre-activation (weighted sum plus bias)
- $\varphi,\ a$: activation function and the neuron's output

## One input: steepness and location

With one input and a sigmoid, factor out $w$ and the two parameters separate cleanly:

$$
\sigma(wx+b)=\sigma\bigl(w\,(x-x^\ast)\bigr),\qquad x^\ast=-\frac{b}{w},\qquad \left.\frac{d a}{d x}\right|_{x=x^\ast}=\frac{w}{4}
$$

- $x^\ast$: the threshold, where $z=0$ and $a=0.5$
- $w/4$: slope of the curve at the threshold, because $\sigma'(0)=1/4$

So $|w|$ sets **how sharp** the transition is and the sign of $w$ sets **which way** it faces. The bias, divided by the weight, sets **where** it happens. At $x=0$ the output is $\sigma(b)$, which is set by the bias alone. A positive bias makes the neuron fire with no input at all.

## Two inputs: a line and its normal

With two inputs, $z=0$ is a line. The weight vector is **perpendicular** to it and points to the side where $z>0$. The bias moves the line along $w$ without turning it.

$$
\{x : w^\top x+b=0\},\qquad \text{distance from origin}=\frac{|b|}{\lVert w\rVert},\qquad \text{signed distance of } x=\frac{w^\top x+b}{\lVert w\rVert}
$$

- $w$: normal vector of the boundary; its direction fixes the line's orientation
- $\lVert w\rVert$: length of w; larger means a sharper sigmoid across the line
- $|b|/\lVert w\rVert$: how far the line sits from the origin

Scaling $(w,b)$ by any $c>0$ leaves the line where it is but multiplies $z$ by $c$. Direction and offset place the boundary. Overall scale sets how confident the neuron is near it.

> Figure: Left: move w to sharpen or flip the sigmoid, and b to slide the threshold (orange). The red dot is the output at x = 0, which depends on b alone. Right: w₁ and w₂ turn the line and set the violet normal. b slides the line along the normal. Press "remove bias": the line is pinned to the origin and no rotation gets below 4 errors, because the two classes lie on the same rays from (0, 0).

**Why the bias is not optional.** Without $b$, every boundary passes through the origin and the input $x=0$ always gives $z=0$ (sigmoid output $0.5$, ReLU output $0$). A bias-free ReLU network is also **positively homogeneous**: $f(cx)=c\,f(x)$ for $c>0$. It can never output a constant offset, and it assigns $cx$ the same class as $x$ for every $c>0$.

## Worked example

One sigmoid neuron with $w=(2,-1)$ and $b=0.5$. The boundary is $2x_1-x_2+0.5=0$, at distance $0.5/\sqrt5\approx0.224$ from the origin.

| input x | w·x | z = w·x + b | a = σ(z) |
| --- | --- | --- | --- |
| (1, 3) | 2 − 3 = −1 | −0.5 | 0.3775 |
| (0, 0) | 0 | 0.5 | 0.6225 |
| (2, 1) | 4 − 1 = 3 | 3.5 | 0.9707 |

The zero input still gives $a=0.62$: the neuron "fires" from its bias alone. In one dimension, $w=2$, $b=-3$ puts the threshold at $x^\ast=1.5$ with slope $0.5$ there. The outputs at $x=1$ and $x=2$ are $0.269$ and $0.731$.

## In a layer: a matrix and a vector

A layer of $n_{out}$ neurons stacks the weight vectors as the rows of a matrix and the biases into a vector. A layer from 784 inputs to 256 units has $256\times784=200{,}704$ weights and $256$ biases, so $200{,}960$ parameters.

$$
z = Wx+b = \begin{bmatrix} W & b\end{bmatrix}\begin{bmatrix} x\\ 1\end{bmatrix},\qquad W\in\mathbb{R}^{n_{out}\times n_{in}},\ b\in\mathbb{R}^{n_{out}}
$$

- $W_{ji}$: weight from input $i$ to neuron $j$ (row $j$ is neuron $j$'s $w$)
- $b_j$: bias of neuron $j$
- $[\,W\ b\,]$: the bias is just the weight on an extra input fixed at 1

The augmented form shows that a bias is a weight whose input is always 1. Gradients treat it that way too: $\partial L/\partial W_{ji}=\delta_j x_i$ and $\partial L/\partial b_j=\delta_j\cdot1$, where $\delta_j=\partial L/\partial z_j$. For a mini-batch, $Z=XW^\top+b$ adds the same $b$ to every row by broadcasting, and the bias gradient is $\delta$ summed over the batch.

```python
import torch, torch.nn as nn

layer = nn.Linear(784, 256)
print(layer.weight.shape, layer.bias.shape)   # (256, 784)  (256,)  - stored (out, in)
x = torch.randn(32, 784)
z = layer(x)                                   # x @ W.T + b  ->  (32, 256)

nn.init.zeros_(layer.bias)                     # common choice; PyTorch's default is U(±1/sqrt(784))

# weight decay on weights only, not on biases (or norm parameters)
decay    = [p for n, p in model.named_parameters() if p.ndim > 1]
no_decay = [p for n, p in model.named_parameters() if p.ndim <= 1]
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.01},
                         {"params": no_decay, "weight_decay": 0.0}], lr=3e-4)
```

## Initialising and regularising them

**Weights must start random**, otherwise every neuron in a layer computes the same thing and gets the same gradient (see weight initialization). **Biases can start at 0**, since random weights already break the symmetry. That is the usual choice in hand-written inits. PyTorch's `nn.Linear` default draws biases from $U(-1/\sqrt{n_{in}},1/\sqrt{n_{in}})$ instead. A useful exception is the output bias: set it to the log-odds of the class prior, $b=\log\frac{p}{1-p}$, so an imbalanced classifier starts out predicting the base rate.

**Biases are usually exempt from weight decay.** An L2 penalty on $w$ limits how sensitive the output is to the input, which is the capacity you want to control. A bias does not change sensitivity, only position. Decaying it just pulls each threshold toward the origin. There are few biases, so they add little capacity anyway.

> **A bias right before BatchNorm does nothing:** BatchNorm subtracts the batch mean, which cancels any constant added just before it, and then adds its own learned shift $\beta$. So `nn.Conv2d(..., bias=False)` followed by `nn.BatchNorm2d` is standard. Keeping the bias wastes parameters and gives the optimizer a direction with zero gradient.

> **Two jobs, two kinds of parameter:** Weights decide **what** a neuron responds to (the direction of $w$) and **how sharply** (its length). The bias decides **where** along that direction the response turns on. Every boundary a network draws is built from those two moves.

**Question:** A sigmoid neuron has $w=4$ and $b=-2$. At which input $x$ is its output exactly $0.5$?

- A. −2
- B. 0.5
- C. 2
- D. −0.5

**Answer:** B. 0.5 — The output is $0.5$ when $z=wx+b=0$, i.e. $x^\ast=-b/w=2/4=0.5$. The slope there is $w/4=1$.

**Question:** What is the shape of `nn.Linear(100, 10).weight`, and how many parameters does the layer have?

- A. (100, 10), 1000
- B. (10, 100), 1010
- C. (10, 100), 1100
- D. (100, 10), 1010

**Answer:** B. (10, 100), 1010 — PyTorch stores the weight as (out, in) = (10, 100) and computes $xW^\top+b$. That is $1000$ weights plus $10$ biases, $1010$ parameters.

**Key points**

- $z=w^\top x+b$: weights scale inputs, the bias shifts the result.
- 1-D: $|w|$ = steepness, $x^\ast=-b/w$ = location of the transition.
- 2-D: $w$ is the boundary's normal, and the line sits $|b|/\lVert w\rVert$ from the origin.
- Without a bias every boundary passes through the origin.
- A layer has $W\in\mathbb{R}^{n_{out}\times n_{in}}$ and $b\in\mathbb{R}^{n_{out}}$. The bias is the weight on a constant 1.
- Biases usually start at 0 and are excluded from weight decay. Drop them right before BatchNorm.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
