Bayesian regression

Put a prior on the weights and return a posterior distribution, so every prediction comes with its own uncertainty.

17 min read1449 wordsadvanced4 viewsupdated 2026-10-07

Definition

Bayesian linear regression treats the weights ww of a linear model as random variables. It starts from a prior belief p(w)p(w), multiplies by the likelihood of the data, and returns the posterior p(w∣data)p(w\mid\text{data}) — a whole distribution over lines. Predictions average over that distribution, so each one comes with a mean and a variance that grows where the data are thin.

At a glance

Task
Regression (real-valued target)
Input → output
features xx → a Gaussian predictive distribution N(μ(x),σ2(x))\mathcal N(\mu(x),\sigma^2(x)), not just a number
Learns
posterior mean mm and covariance SS of the weights; optionally the prior precision α\alpha and noise precision β\beta
Training
O(NM2+M3)O(NM^2+M^3) for NN rows and MM features (one M×MM\times M inverse); evidence maximization repeats a cheap update a few times
Prediction
mean O(M)O(M), variance O(M2)O(M^2) per point
Hyperparameters
prior precision α\alpha, noise precision β\beta (or the Gamma hyperpriors on them), the basis functions ϕ(x)\phi(x)
Assumes
linear in the features, Gaussian noise with constant variance, Gaussian prior on the weights

The problem

Ordinary least squares (see Linear regression and Multiple linear regression) returns one line: the single w^\hat w that minimizes squared error. It says nothing about how sure it is. Ask it to predict at x=2x=2, right in the middle of the training data, and at x=6x=6, far beyond it, and it answers both with the same confidence. With six points, those two answers deserve very different trust.

OLS has a second problem with small data: it overfits. Ridge regression fixes that with a penalty λ∥w∥2\lambda\lVert w\rVert^2, but λ\lambda looks arbitrary and still has to be tuned by cross-validation, which is noisy when NN is tiny. Bayesian regression explains where the penalty comes from (a prior), gives error bars for free, and can pick λ\lambda from the training data alone by maximizing the evidence.

The idea

Before seeing data, say which lines are plausible: slopes and intercepts near zero, spread out with variance 1/α1/\alpha. Each data point then rules out lines that pass far from it — in weight space, a point (xn,yn)(x_n,y_n) is compatible only with the strip of (w0,w1)(w_0,w_1) where w0+w1xn≈ynw_0+w_1x_n\approx y_n. Multiply the prior by all those strips and what survives is the posterior: an ellipse of lines that fit the data and are not too extreme. To predict, ask every line in the ellipse and report the average and the spread of their answers.

Vocabulary

Prior
p(w)=N(0,α−1I)p(w)=\mathcal N(0,\alpha^{-1}I): belief about the weights before seeing data. Large α\alpha = confident that weights are small.
Likelihood
p(y∣X,w)p(y\mid X,w): how probable the observed targets are if the weights were ww. Gaussian noise makes it ∏nN(yn∣w⊤ϕn,β−1)\prod_n\mathcal N(y_n\mid w^\top\phi_n,\beta^{-1}).
Posterior
p(w∣X,y)∝p(w\mid X,y)\propto likelihood × prior. For this model it is again Gaussian, N(m,S)\mathcal N(m,S).
Precision
inverse variance. α\alpha = prior precision of the weights; β\beta = precision of the noise (σ2=1/β\sigma^2=1/\beta).
Conjugate prior
a prior whose posterior has the same family. Gaussian prior + Gaussian likelihood → Gaussian posterior, so everything has a closed form.
Posterior predictive
p(y∗∣x∗,data)=∫p(y∗∣x∗,w) p(w∣data) dwp(y^*\mid x^*,\text{data})=\int p(y^*\mid x^*,w)\,p(w\mid\text{data})\,dw: the forecast averaged over all plausible weights.
MAP estimate
the single most probable ww (the posterior mode). Here it equals the posterior mean, and equals ridge regression.
Evidence (marginal likelihood)
p(y∣X,α,β)p(y\mid X,\alpha,\beta): probability of the data with the weights integrated out. Used to choose α,β\alpha,\beta — "empirical Bayes" or "type-II maximum likelihood".
Epistemic vs aleatoric
uncertainty about ww (shrinks with more data) vs irreducible noise 1/β1/\beta (does not).

The math

Stack the inputs into a design matrix Φ\Phi whose row nn is ϕ(xn)⊤\phi(x_n)^\top — for a straight line, ϕ(x)=(1,x)\phi(x)=(1,x); for polynomial or RBF features, any fixed basis. The model and prior are:

yn=w⊤ϕ(xn)+εn,εn∼N(0,β−1),p(w)=N(w∣0, α−1I)y_n = w^\top\phi(x_n)+\varepsilon_n,\quad \varepsilon_n\sim\mathcal N(0,\beta^{-1}),\qquad p(w)=\mathcal N(w\mid 0,\ \alpha^{-1}I)
w∈RMw\in\mathbb R^M
weight vector (including the intercept, if φ has a constant 1)
ϕ(xn)\phi(x_n)
feature vector of row n; Φ is the N×M matrix of all of them
β\beta
noise precision: noise variance is 1/β
α\alpha
prior precision: each weight has prior variance 1/α

Derivation of the posterior. Take logs of likelihood × prior and drop everything that does not depend on ww:

ln⁡p(w∣y)=−β2∥y−Φw∥2−α2w⊤w+const=−12w⊤ ⁣(αI+βΦ⊤Φ)w+β w⊤Φ⊤y+const\ln p(w\mid y)= -\frac{\beta}{2}\lVert y-\Phi w\rVert^2-\frac{\alpha}{2}w^\top w+\text{const} = -\frac12 w^\top\!\big(\alpha I+\beta\Phi^\top\Phi\big)w+\beta\,w^\top\Phi^\top y+\text{const}
∥y−Φw∥2\lVert y-\Phi w\rVert^2
sum of squared residuals — the OLS loss
w⊤ww^\top w
squared L2 norm — the ridge penalty

This is a quadratic in ww, so the posterior is Gaussian. A Gaussian N(m,S)\mathcal N(m,S) has log-density −12w⊤S−1w+w⊤S−1m+const-\tfrac12 w^\top S^{-1}w+w^\top S^{-1}m+\text{const}. Matching the quadratic term gives the precision; matching the linear term gives the mean:

S−1=A=αI+β Φ⊤Φ,m=β S Φ⊤yS^{-1}=A=\alpha I+\beta\,\Phi^\top\Phi,\qquad m=\beta\,S\,\Phi^\top y
AA
posterior precision: prior precision plus β times the data's information Φᵀ Φ
SS
posterior covariance of the weights
mm
posterior mean (= mode = MAP estimate)

Divide inside by β\beta: m=(Φ⊤Φ+αβI)−1Φ⊤ym=(\Phi^\top\Phi+\tfrac{\alpha}{\beta}I)^{-1}\Phi^\top y. That is exactly ridge regression with λ=α/β\lambda=\alpha/\beta — the ridge penalty is a Gaussian prior, and its strength is the ratio prior precision / noise precision. As α→0\alpha\to0 (a flat prior) the mean becomes OLS.

The predictive distribution. A new target is y∗=w⊤ϕ∗+εy^*=w^\top\phi^*+\varepsilon with ϕ∗=ϕ(x∗)\phi^*=\phi(x^*). Under the posterior, w⊤ϕ∗w^\top\phi^* is a linear function of a Gaussian, so it is Gaussian with mean m⊤ϕ∗m^\top\phi^* and variance ϕ∗⊤Sϕ∗\phi^{*\top}S\phi^*; the noise ε\varepsilon is independent, so the variances add:

p(y∗∣x∗,data)=N ⁣(y∗ ∣ m⊤ϕ∗,  σ2(x∗)=1β⏟noise+ϕ∗⊤S ϕ∗⏟weight uncertainty)p(y^*\mid x^*,\text{data})=\mathcal N\!\Big(y^*\ \Big|\ m^\top\phi^*,\ \ \sigma^2(x^*)=\underbrace{\tfrac{1}{\beta}}_{\text{noise}}+\underbrace{\phi^{*\top}S\,\phi^*}_{\text{weight uncertainty}}\Big)
1/β1/\beta
aleatoric term: the noise floor, the same everywhere
ϕ∗⊤Sϕ∗\phi^{*\top}S\phi^*
epistemic term: large where x* is far from the training inputs, shrinks like 1/N

Sequential learning. Because A=αI+β∑nϕnϕn⊤A=\alpha I+\beta\sum_n\phi_n\phi_n^\top is a sum over points, adding one observation just adds βϕnϕn⊤\beta\phi_n\phi_n^\top to the precision: yesterday's posterior is today's prior. The order of the points does not matter, and you can update online without storing old data.

Choosing α and β from the data: the evidence. Integrating ww out of likelihood × prior (a Gaussian integral) gives the log marginal likelihood:

ln⁡p(y∣α,β)=M2ln⁡α+N2ln⁡β−E(m)−12ln⁡∣A∣−N2ln⁡2π,E(m)=β2∥y−Φm∥2+α2m⊤m\ln p(y\mid\alpha,\beta)=\frac M2\ln\alpha+\frac N2\ln\beta-E(m)-\frac12\ln|A|-\frac N2\ln 2\pi,\qquad E(m)=\frac\beta2\lVert y-\Phi m\rVert^2+\frac\alpha2 m^\top m
E(m)E(m)
data misfit plus weight size at the posterior mean (the ridge objective)
12ln⁡∣A∣\tfrac12\ln|A|
Occam penalty: grows when the data pin the weights down far more tightly than the prior did

Set the derivatives to zero. With λi\lambda_i the eigenvalues of βΦ⊤Φ\beta\Phi^\top\Phi, ∂/∂α\partial/\partial\alpha gives M2α−12m⊤m−12∑i1α+λi=0\tfrac{M}{2\alpha}-\tfrac12m^\top m-\tfrac12\sum_i\tfrac1{\alpha+\lambda_i}=0; rearranging gives MacKay's fixed-point updates:

γ=∑i=1Mλiα+λi,α←γm⊤m,β←N−γ∥y−Φm∥2\gamma=\sum_{i=1}^{M}\frac{\lambda_i}{\alpha+\lambda_i},\qquad \alpha\leftarrow\frac{\gamma}{m^\top m},\qquad \beta\leftarrow\frac{N-\gamma}{\lVert y-\Phi m\rVert^2}
γ\gamma
effective number of parameters, between 0 and M: how many weight directions the data actually determine
N−γN-\gamma
degrees of freedom left for estimating the noise (like N − p in the unbiased variance)

Alternate: compute m,Sm,S with the current α,β\alpha,\beta, update α,β\alpha,\beta, repeat. This is what scikit-learn's BayesianRidge does (it also adds tiny Gamma hyperpriors, 10−610^{-6}, so nothing divides by zero).

What each hyperparameter does

α ↑ (stronger prior)
adds αI\alpha I to AA: shrinks mm toward 0 and narrows the band. Too large and the model is confidently wrong (underfit).
α → 0
flat prior: m→m\to OLS, and S→(βΦ⊤Φ)−1S\to(\beta\Phi^\top\Phi)^{-1}, the classical OLS covariance.
β ↑ (less noise)
trusts each point more: the likelihood strips get thinner, the posterior tightens, and the noise floor 1/β1/\beta drops.
α/β
the only thing the posterior mean depends on — the ridge λ\lambda. The band depends on both separately.

Worked example: six weeks of ad spend

A small shop records weekly ad spend xx (hundreds of dollars) and sales yy (thousands of units). We fix the prior precision α=1\alpha=1 (weights have prior sd 1) and the noise precision β=4\beta=4 (noise sd 0.5), use ϕ(x)=(1,x)\phi(x)=(1,x), and want forecasts at x∗=2x^*=2 (inside the data) and x∗=6x^*=6 (an extrapolation).

Weekx (ad spend)y (sales)
10.51.2
21.02.0
31.52.1
42.53.1
53.03.2
64.04.4
Six observations. ∑x=12.5\sum x=12.5, ∑x2=34.75\sum x^2=34.75, ∑y=16.0\sum y=16.0, ∑xy=40.7\sum xy=40.7.

The posterior and two forecasts, by hand

  1. 1Data information. Φ⊤Φ=(N∑x∑x∑x2)=(612.512.534.75)\Phi^\top\Phi=\begin{pmatrix}N&\sum x\\\sum x&\sum x^2\end{pmatrix}=\begin{pmatrix}6&12.5\\12.5&34.75\end{pmatrix} and Φ⊤y=(∑y∑xy)=(16.040.7)\Phi^\top y=\begin{pmatrix}\sum y\\\sum xy\end{pmatrix}=\begin{pmatrix}16.0\\40.7\end{pmatrix}.
  2. 2Posterior precision. A=αI+βΦ⊤Φ=(1+2450501+139)=(255050140)A=\alpha I+\beta\Phi^\top\Phi=\begin{pmatrix}1+24&50\\50&1+139\end{pmatrix}=\begin{pmatrix}25&50\\50&140\end{pmatrix}, with ∣A∣=25⋅140−502=1000|A|=25\cdot140-50^2=1000.
  3. 3Posterior covariance. S=A−1=11000(140−50−5025)=(0.14−0.05−0.050.025)S=A^{-1}=\tfrac1{1000}\begin{pmatrix}140&-50\\-50&25\end{pmatrix}=\begin{pmatrix}0.14&-0.05\\-0.05&0.025\end{pmatrix}.
  4. 4Posterior mean. βΦ⊤y=(64, 162.8)\beta\Phi^\top y=(64,\ 162.8), so m=S (64,162.8)⊤=(0.14⋅64−0.05⋅162.8, −0.05⋅64+0.025⋅162.8)=(0.82, 0.87)m=S\,(64,162.8)^\top=(0.14\cdot64-0.05\cdot162.8,\ -0.05\cdot64+0.025\cdot162.8)=(\mathbf{0.82},\ \mathbf{0.87}).
  5. 5Uncertainty in the weights. sd⁡(w0)=0.14=0.374\operatorname{sd}(w_0)=\sqrt{0.14}=0.374, sd⁡(w1)=0.025=0.158\operatorname{sd}(w_1)=\sqrt{0.025}=0.158, correlation −0.05/0.14⋅0.025=−0.845-0.05/\sqrt{0.14\cdot0.025}=-0.845: a steeper slope must come with a lower intercept to pass through the same points.
  6. 6*Forecast at $x^=2.∗∗Mean.** Mean 0.82+0.87\cdot2=2.56.Weightterm. Weight term \phi^\top S\phi=0.14+2\cdot2\cdot(-0.05)+4\cdot0.025=0.04.Variance. Variance 0.25+0.04=0.29,sd, sd =0.539$ — barely above the noise floor of 0.5.
  7. 7*Forecast at $x^=6.∗∗Mean.** Mean 0.82+0.87\cdot6=6.04.Weightterm. Weight term 0.14-0.6+0.9=0.44.Variance. Variance 0.25+0.44=0.69,sd, sd =\mathbf{0.831}.The95. The 95% predictive interval is 6.04\pm1.96\cdot0.831=[4.41,\ 7.67]$.
  8. 8Compare. OLS gives (0.904, 0.846)(0.904,\ 0.846); ridge with λ=α/β=0.25\lambda=\alpha/\beta=0.25 (penalizing the intercept too) gives exactly (0.82, 0.87)(0.82,\ 0.87) — the posterior mean. Bayes adds the band.

Now feed the same six points in one at a time, starting from the prior. Each row is the posterior after nn points — notice how the slope's sd and the forecast sd at x∗=6x^*=6 collapse, but never below the noise floor 1/β=0.5\sqrt{1/\beta}=0.5.

n seenmean (w₀, w₁)sd(w₀)sd(w₁)forecast sd at x* = 6
0 (prior)(0, 0)1.0001.0006.103
1(0.800, 0.400)0.5770.9135.156
2(0.800, 0.933)0.5770.7073.819
3(0.824, 0.875)0.5420.5052.626
4(0.796, 0.912)0.4520.2951.493
5(0.875, 0.825)0.4160.2181.102
6(0.820, 0.870)0.3740.1580.831
Sequential updating with α=1, β=4\alpha=1,\ \beta=4. Row n=6n=6 is the batch answer above.
weights · - - prior · posterior-1-1001122w₀ (intercept)w₁ (slope)newest point's likelihood× OLSdata · mean ± 1σ, 2σ · drag0123456-202468x (ad spend)x* = 6each coloured dot (left) is one posterior sample = one coloured line (right)
posterior mean w = (0.820, 0.870)
posterior sd = (0.374, 0.158) · corr -0.85
predict x* = 6: 6.04 ± 0.831
= noise 0.250 + weights 0.440 (variance)
same as ridge with λ = α/β = 0.25
log evidence ln p(y|α,β) = -5.871
fig.Left: weight space — the dashed prior circles (α = 1), the posterior 1σ and 2σ ellipses, the newest point's likelihood strip (orange) and the OLS estimate (×). Right: the same posterior as lines — mean, ±1σ/±2σ predictive band and five posterior samples (each coloured dot on the left is one coloured line on the right). Press play to add the six points one by one, or drag n, α, β or any point. At the defaults the readout matches the worked example: mean (0.820, 0.870), forecast at x* = 6 of 6.04 ± 0.831, ridge λ = 0.25.
αmean (w₀, w₁)forecast sd at x* = 6log evidence
0.01(0.903, 0.846)0.855−9.656
0.1(0.895, 0.849)0.853−7.431
1(0.820, 0.870)0.831−5.871
4(0.673, 0.903)0.785−6.695
10(0.544, 0.910)0.742−9.610
100(0.264, 0.626)0.633−40.579
Effect of the prior precision α on the same data (β = 4). The log evidence peaks near α ≈ 1.28 (value −5.845), so α = 1 is close to what the data prefer; α = 100 squashes the line toward zero and makes the band narrower — confidently wrong.

The algorithm, step by step

Bayesian linear regression with evidence maximization

  1. 1Features. Build Φ\Phi from ϕ(x)\phi(x). If you will not put a prior on the intercept, centre XX and yy first (this is what scikit-learn does) and add the intercept back at the end.
  2. 2Initialize α\alpha (e.g. 1) and β\beta (e.g. 1/Var⁡(y)1/\operatorname{Var}(y)). Precompute the eigenvalues of Φ⊤Φ\Phi^\top\Phi (or an SVD of Φ\Phi) once.
  3. 3Posterior. A=αI+βΦ⊤ΦA=\alpha I+\beta\Phi^\top\Phi, S=A−1S=A^{-1}, m=βSΦ⊤ym=\beta S\Phi^\top y. (With the SVD, this is cheap for every new α,β\alpha,\beta.)
  4. 4Re-estimate γ=∑iβei/(α+βei)\gamma=\sum_i\beta e_i/(\alpha+\beta e_i), α=γ/m⊤m\alpha=\gamma/m^\top m, β=(N−γ)/∥y−Φm∥2\beta=(N-\gamma)/\lVert y-\Phi m\rVert^2.
  5. 5Repeat steps 3–4 until mm stops changing (typically a handful of iterations). Skip 2–4 entirely if you have fixed α,β\alpha,\beta from domain knowledge.
  6. 6Predict. For each new x∗x^*: mean m⊤ϕ∗m^\top\phi^*, variance 1/β+ϕ∗⊤Sϕ∗1/\beta+\phi^{*\top}S\phi^*; report e.g. mean ±1.96\pm1.96 sd.

Complexity. Forming Φ⊤Φ\Phi^\top\Phi costs O(NM2)O(NM^2) and the inverse or eigendecomposition O(M3)O(M^3) — the same as OLS. With the eigenbasis cached, each evidence iteration is only O(M2)O(M^2). Prediction is O(M)O(M) for the mean and O(M2)O(M^2) for the variance. Memory is O(M2)O(M^2) for SS, independent of NN, and sequential updating never needs the old rows.

Variants and practical tuning

The family

BayesianRidge (sklearn)
one shared prior precision for all weights, learned by evidence maximization. Posterior mean ≈ ridge with a data-chosen λ\lambda.
ARD (ARDRegression)
automatic relevance determination: one αj\alpha_j per weight. Irrelevant features get αj→∞\alpha_j\to\infty, so their weights are pruned to 0 — a Bayesian route to sparsity, like lasso.
Unknown noise, conjugate
a Normal–Inverse-Gamma prior on (w,σ2)(w,\sigma^2) integrates the noise out analytically; the predictive becomes a Student-tt with heavier tails for small NN.
Basis functions
replace (1,x)(1,x) with polynomial or RBF features (see Polynomial regression); all formulas are unchanged, and the band now follows the curve.
Gaussian processes
the kernelized limit: put the prior directly on functions. Bayesian linear regression with MM basis functions is a GP with kernel k(x,x′)=α−1ϕ(x)⊤ϕ(x′)k(x,x')=\alpha^{-1}\phi(x)^\top\phi(x').
Non-conjugate models
robust (Student-t) noise, hierarchical priors, Bayesian logistic regression: no closed form — use Laplace approximation, variational inference or MCMC (PyMC, Stan, NumPyro).
ParameterDefaultMeaning / when to change
alpha_1, alpha_21e-6Gamma hyperprior (shape, rate) on the noise precision; raise to encode a belief about the noise level
lambda_1, lambda_21e-6Gamma hyperprior on the weight precision; raise lambda_1/lambda_2 to force more shrinkage
alpha_init, lambda_init1/Var(y), 1starting values of the evidence iterations
max_iter, tol300, 1e-3stop when the coefficients change by less than tol
fit_interceptTruecentres X and y; the intercept gets no prior and no uncertainty
compute_scoreFalsestore the log marginal likelihood at every iteration in scores_
scikit-learn BayesianRidge knobs. Note the naming: sklearn's alpha is the noise precision (our β) and its lambda is the weight precision (our α).

In code

python
1import numpy as np
2
3def posterior(Phi, y, alpha, beta):
4 """Gaussian posterior N(m, S) over weights; prior N(0, alpha^-1 I), noise precision beta."""
5 A = alpha * np.eye(Phi.shape[1]) + beta * Phi.T @ Phi # posterior precision
6 S = np.linalg.inv(A)
7 m = beta * S @ Phi.T @ y
8 return m, S
9
10def predict(Phi_new, m, S, beta):
11 mean = Phi_new @ m
12 var = 1 / beta + np.einsum("ij,jk,ik->i", Phi_new, S, Phi_new)
13 return mean, np.sqrt(var)
14
15def log_evidence(Phi, y, alpha, beta):
16 N, M = Phi.shape
17 m, S = posterior(Phi, y, alpha, beta)
18 E = beta / 2 * np.sum((y - Phi @ m) ** 2) + alpha / 2 * m @ m
19 return (M / 2 * np.log(alpha) + N / 2 * np.log(beta) - E
20 - 0.5 * np.linalg.slogdet(np.linalg.inv(S))[1] - N / 2 * np.log(2 * np.pi))
21
22def fit_evidence(X, y, iters=300, tol=1e-3):
23 """Empirical Bayes (MacKay) on centred data, like sklearn's BayesianRidge."""
24 xm, ym = X.mean(0), y.mean()
25 Xc, yc = X - xm, y - ym
26 N = len(y)
27 eig = np.linalg.eigvalsh(Xc.T @ Xc)
28 alpha, beta = 1.0, 1 / (np.var(y) + 1e-12) # weight precision, noise precision
29 w = np.zeros(X.shape[1])
30 for _ in range(iters):
31 m, S = posterior(Xc, yc, alpha, beta)
32 gamma = np.sum(beta * eig / (alpha + beta * eig)) # effective no. of parameters
33 alpha = (gamma + 2e-6) / (m @ m + 2e-6) # tiny Gamma hyperpriors, as sklearn
34 beta = (N - gamma + 2e-6) / (np.sum((yc - Xc @ m) ** 2) + 2e-6)
35 if np.abs(m - w).sum() < tol:
36 break
37 w = m
38 m, S = posterior(Xc, yc, alpha, beta)
39 return m, ym - xm @ m, S, alpha, beta, gamma, xm
40
41x = np.array([0.5, 1.0, 1.5, 2.5, 3.0, 4.0])
42y = np.array([1.2, 2.0, 2.1, 3.1, 3.2, 4.4])
43Phi = np.c_[np.ones_like(x), x]
44
45m, S = posterior(Phi, y, alpha=1.0, beta=4.0)
46print("posterior mean", m.round(3), "sd", np.sqrt(np.diag(S)).round(3))
47mean, sd = predict(np.array([[1, 2.0], [1, 6.0]]), m, S, beta=4.0)
48print("predict x=2, 6:", mean.round(3), "+/-", sd.round(3))
49print("log evidence (alpha=1, beta=4):", round(log_evidence(Phi, y, 1.0, 4.0), 3))
50
51w, b, S, alpha, beta, gamma, xm = fit_evidence(x[:, None], y)
52print(f"evidence fit: w={w[0]:.4f} b={b:.4f} alpha={alpha:.4f} beta={beta:.4f} gamma={gamma:.4f}")
53Xn = np.array([[2.0], [6.0]])
54print("predict:", (Xn @ w + b).round(3), "+/-", np.sqrt(1 / beta + np.einsum("ij,jk,ik->i", Xn, S, Xn)).round(3))
55Xc = Xn - xm # honest band: intercept uncertainty 1/(N beta), leverage measured from the data mean
56print("centred band:", np.sqrt(1 / beta + 1 / (len(y) * beta) + np.einsum("ij,jk,ik->i", Xc, S, Xc)).round(3))
57
58# posterior mean [0.82 0.87] sd [0.374 0.158]
59# predict x=2, 6: [2.56 6.04] +/- [0.539 0.831]
60# log evidence (alpha=1, beta=4): -5.871
61# evidence fit: w=0.8415 b=0.9134 alpha=1.4047 beta=30.9351 gamma=0.9948
62# predict: [2.597 5.963] +/- [0.217 0.407]
63# centred band: [0.194 0.307]
From scratch with NumPy: fixed-α/β posterior (the worked example) and MacKay evidence maximization. Prints mean (0.82, 0.87), forecasts 2.56 ± 0.539 and 6.04 ± 0.831, then the evidence fit that matches scikit-learn below.
python
1import numpy as np
2from sklearn.linear_model import BayesianRidge, LinearRegression
3
4x = np.array([0.5, 1.0, 1.5, 2.5, 3.0, 4.0]).reshape(-1, 1)
5y = np.array([1.2, 2.0, 2.1, 3.1, 3.2, 4.4])
6
7model = BayesianRidge(compute_score=True).fit(x, y)
8print(f"slope {model.coef_[0]:.4f} intercept {model.intercept_:.4f}")
9print(f"noise precision alpha_ {model.alpha_:.4f} (noise sd {model.alpha_ ** -0.5:.4f})")
10print(f"weight precision lambda_ {model.lambda_:.4f}")
11print(f"log evidence {model.scores_[-1]:.4f} after {len(model.scores_) - 1} iterations")
12
13mean, std = model.predict(np.array([[2.0], [6.0]]), return_std=True)
14print("predict x=2, 6:", mean.round(3), "+/-", std.round(3))
15print("OLS slope", LinearRegression().fit(x, y).coef_.round(4))
16
17# slope 0.8415 intercept 0.9134
18# noise precision alpha_ 30.9351 (noise sd 0.1798)
19# weight precision lambda_ 1.4047
20# log evidence -0.8487 after 4 iterations
21# predict x=2, 6: [2.597 5.963] +/- [0.217 0.407]
22# OLS slope [0.8459]
scikit-learn's BayesianRidge on the worked example: α and β are learned by evidence maximization instead of being fixed.

The evidence decides the noise sd is about 0.18, not the 0.5 we assumed — the six points sit closer to a line than our guess — and γ=0.995\gamma=0.995: the data fully determine the one slope. The learned weight precision 1.40 shrinks the slope only slightly, from OLS's 0.8459 to 0.8415. With the tighter noise, the 6.0 forecast is 5.963±0.4075.963\pm0.407. (scikit-learn's scores_ include the hyperprior terms, so they are not on the same scale as our −5.871.)

Use when

  • +You need calibrated error bars per prediction — small data, extrapolation, decisions that depend on risk.
  • +Data arrive one at a time and you want cheap online updates without storing the history.
  • +You want ridge-like regularization but have too few rows to cross-validate λ\lambda reliably — maximize the evidence instead.
  • +You have genuine prior knowledge about plausible coefficient sizes and want to encode it.

Avoid when

  • −The relationship is strongly non-linear and you have no good basis functions — use a GP, trees or boosting.
  • −Noise is heavy-tailed or has outliers: Gaussian likelihood lets one bad point drag the posterior (use Student-t noise via MCMC).
  • −MM is huge (tens of thousands of features): the M×MM\times M covariance is expensive; use variational or diagonal approximations.
  • −You only need a point prediction on a big dataset — the posterior collapses to OLS/ridge and the extra machinery buys little.

Where it's used

Bayesian optimization

Surrogate models (Bayesian linear regression on learned features, or its GP limit) whose uncertainty drives which hyperparameters or experiments to try next.

Marketing mix modelling

Estimating the sales effect of each ad channel from a few years of weekly data, with priors from past campaigns and credible intervals for budget decisions.

Calibration curves in science

Instrument and dose–response calibration where every reading needs an uncertainty attached, and new measurements update the fit online.

Contextual bandits

Thompson sampling with a Bayesian linear model per action: sample weights from the posterior, act greedily on the sample — exploration comes from the uncertainty.

Check yourself

With prior precision α=1\alpha=1 and noise precision β=4\beta=4, the posterior mean equals ridge regression with which λ\lambda?

Check yourself

Keep β=4\beta=4 and collect more and more data across the same xx-range. The predictive sd at x∗=2x^*=2 tends to…

Check yourself

Using the worked example's S=(0.14−0.05−0.050.025)S=\begin{pmatrix}0.14&-0.05\\-0.05&0.025\end{pmatrix} and β=4\beta=4, the predictive variance at x∗=0x^*=0 is…

Check yourself

In scikit-learn's BayesianRidge, the fitted attribute alpha_ is…

Remember

  • ✓Gaussian prior N(0,α−1I)\mathcal N(0,\alpha^{-1}I) × Gaussian likelihood → Gaussian posterior with S−1=αI+βΦ⊤ΦS^{-1}=\alpha I+\beta\Phi^\top\Phi and m=βSΦ⊤ym=\beta S\Phi^\top y.
  • ✓The posterior mean is ridge regression with λ=α/β\lambda=\alpha/\beta; a flat prior (α→0\alpha\to0) gives OLS.
  • ✓Predictive variance =1/β+ϕ⊤Sϕ=1/\beta+\phi^\top S\phi: a constant noise floor plus weight uncertainty that grows away from the data and shrinks like 1/N1/N.
  • ✓Updating is sequential: each point adds βφφᵀ to the precision, and the posterior becomes the next prior.
  • ✓Choose α,β\alpha,\beta by maximizing the evidence (MacKay updates with γ\gamma, the effective number of parameters) — this is what BayesianRidge does.
  • ✓The error bars are only trustworthy if the model is right and features are centred and scaled; check calibration on held-out data.
← back to ML Algorithms roadmap