Definition
Bayesian linear regression treats the weights of a linear model as random variables. It starts from a prior belief , multiplies by the likelihood of the data, and returns the posterior — a whole distribution over lines. Predictions average over that distribution, so each one comes with a mean and a variance that grows where the data are thin.
At a glance
- Task
- Regression (real-valued target)
- Input → output
- features → a Gaussian predictive distribution , not just a number
- Learns
- posterior mean and covariance of the weights; optionally the prior precision and noise precision
- Training
- for rows and features (one inverse); evidence maximization repeats a cheap update a few times
- Prediction
- mean , variance per point
- Hyperparameters
- prior precision , noise precision (or the Gamma hyperpriors on them), the basis functions
- Assumes
- linear in the features, Gaussian noise with constant variance, Gaussian prior on the weights
The problem
Ordinary least squares (see Linear regression and Multiple linear regression) returns one line: the single that minimizes squared error. It says nothing about how sure it is. Ask it to predict at , right in the middle of the training data, and at , far beyond it, and it answers both with the same confidence. With six points, those two answers deserve very different trust.
OLS has a second problem with small data: it overfits. Ridge regression fixes that with a penalty , but looks arbitrary and still has to be tuned by cross-validation, which is noisy when is tiny. Bayesian regression explains where the penalty comes from (a prior), gives error bars for free, and can pick from the training data alone by maximizing the evidence.
The idea
Before seeing data, say which lines are plausible: slopes and intercepts near zero, spread out with variance . Each data point then rules out lines that pass far from it — in weight space, a point is compatible only with the strip of where . Multiply the prior by all those strips and what survives is the posterior: an ellipse of lines that fit the data and are not too extreme. To predict, ask every line in the ellipse and report the average and the spread of their answers.
Vocabulary
- Prior
- : belief about the weights before seeing data. Large = confident that weights are small.
- Likelihood
- : how probable the observed targets are if the weights were . Gaussian noise makes it .
- Posterior
- likelihood × prior. For this model it is again Gaussian, .
- Precision
- inverse variance. = prior precision of the weights; = precision of the noise ().
- Conjugate prior
- a prior whose posterior has the same family. Gaussian prior + Gaussian likelihood → Gaussian posterior, so everything has a closed form.
- Posterior predictive
- : the forecast averaged over all plausible weights.
- MAP estimate
- the single most probable (the posterior mode). Here it equals the posterior mean, and equals ridge regression.
- Evidence (marginal likelihood)
- : probability of the data with the weights integrated out. Used to choose — "empirical Bayes" or "type-II maximum likelihood".
- Epistemic vs aleatoric
- uncertainty about (shrinks with more data) vs irreducible noise (does not).
The math
Stack the inputs into a design matrix whose row is — for a straight line, ; for polynomial or RBF features, any fixed basis. The model and prior are:
- weight vector (including the intercept, if φ has a constant 1)
- feature vector of row n; Φ is the N×M matrix of all of them
- noise precision: noise variance is 1/β
- prior precision: each weight has prior variance 1/α
Derivation of the posterior. Take logs of likelihood × prior and drop everything that does not depend on :
- sum of squared residuals — the OLS loss
- squared L2 norm — the ridge penalty
This is a quadratic in , so the posterior is Gaussian. A Gaussian has log-density . Matching the quadratic term gives the precision; matching the linear term gives the mean:
- posterior precision: prior precision plus β times the data's information Φᵀ Φ
- posterior covariance of the weights
- posterior mean (= mode = MAP estimate)
Divide inside by : . That is exactly ridge regression with — the ridge penalty is a Gaussian prior, and its strength is the ratio prior precision / noise precision. As (a flat prior) the mean becomes OLS.
The predictive distribution. A new target is with . Under the posterior, is a linear function of a Gaussian, so it is Gaussian with mean and variance ; the noise is independent, so the variances add:
- aleatoric term: the noise floor, the same everywhere
- epistemic term: large where x* is far from the training inputs, shrinks like 1/N
Sequential learning. Because is a sum over points, adding one observation just adds to the precision: yesterday's posterior is today's prior. The order of the points does not matter, and you can update online without storing old data.
Choosing α and β from the data: the evidence. Integrating out of likelihood × prior (a Gaussian integral) gives the log marginal likelihood:
- data misfit plus weight size at the posterior mean (the ridge objective)
- Occam penalty: grows when the data pin the weights down far more tightly than the prior did
Set the derivatives to zero. With the eigenvalues of , gives ; rearranging gives MacKay's fixed-point updates:
- effective number of parameters, between 0 and M: how many weight directions the data actually determine
- degrees of freedom left for estimating the noise (like N − p in the unbiased variance)
Alternate: compute with the current , update , repeat. This is what scikit-learn's BayesianRidge does (it also adds tiny Gamma hyperpriors, , so nothing divides by zero).
What each hyperparameter does
- α ↑ (stronger prior)
- adds to : shrinks toward 0 and narrows the band. Too large and the model is confidently wrong (underfit).
- α → 0
- flat prior: OLS, and , the classical OLS covariance.
- β ↑ (less noise)
- trusts each point more: the likelihood strips get thinner, the posterior tightens, and the noise floor drops.
- α/β
- the only thing the posterior mean depends on — the ridge . The band depends on both separately.
Worked example: six weeks of ad spend
A small shop records weekly ad spend (hundreds of dollars) and sales (thousands of units). We fix the prior precision (weights have prior sd 1) and the noise precision (noise sd 0.5), use , and want forecasts at (inside the data) and (an extrapolation).
| Week | x (ad spend) | y (sales) |
|---|---|---|
| 1 | 0.5 | 1.2 |
| 2 | 1.0 | 2.0 |
| 3 | 1.5 | 2.1 |
| 4 | 2.5 | 3.1 |
| 5 | 3.0 | 3.2 |
| 6 | 4.0 | 4.4 |
The posterior and two forecasts, by hand
- 1Data information. and .
- 2Posterior precision. , with .
- 3Posterior covariance. .
- 4Posterior mean. , so .
- 5Uncertainty in the weights. , , correlation : a steeper slope must come with a lower intercept to pass through the same points.
- 6*Forecast at $x^=20.82+0.87\cdot2=2.56\phi^\top S\phi=0.14+2\cdot2\cdot(-0.05)+4\cdot0.025=0.040.25+0.04=0.29=0.539$ — barely above the noise floor of 0.5.
- 7*Forecast at $x^=60.82+0.87\cdot6=6.040.14-0.6+0.9=0.440.25+0.44=0.69=\mathbf{0.831}6.04\pm1.96\cdot0.831=[4.41,\ 7.67]$.
- 8Compare. OLS gives ; ridge with (penalizing the intercept too) gives exactly — the posterior mean. Bayes adds the band.
Now feed the same six points in one at a time, starting from the prior. Each row is the posterior after points — notice how the slope's sd and the forecast sd at collapse, but never below the noise floor .
| n seen | mean (w₀, w₁) | sd(w₀) | sd(w₁) | forecast sd at x* = 6 |
|---|---|---|---|---|
| 0 (prior) | (0, 0) | 1.000 | 1.000 | 6.103 |
| 1 | (0.800, 0.400) | 0.577 | 0.913 | 5.156 |
| 2 | (0.800, 0.933) | 0.577 | 0.707 | 3.819 |
| 3 | (0.824, 0.875) | 0.542 | 0.505 | 2.626 |
| 4 | (0.796, 0.912) | 0.452 | 0.295 | 1.493 |
| 5 | (0.875, 0.825) | 0.416 | 0.218 | 1.102 |
| 6 | (0.820, 0.870) | 0.374 | 0.158 | 0.831 |
| α | mean (w₀, w₁) | forecast sd at x* = 6 | log evidence |
|---|---|---|---|
| 0.01 | (0.903, 0.846) | 0.855 | −9.656 |
| 0.1 | (0.895, 0.849) | 0.853 | −7.431 |
| 1 | (0.820, 0.870) | 0.831 | −5.871 |
| 4 | (0.673, 0.903) | 0.785 | −6.695 |
| 10 | (0.544, 0.910) | 0.742 | −9.610 |
| 100 | (0.264, 0.626) | 0.633 | −40.579 |
The algorithm, step by step
Bayesian linear regression with evidence maximization
- 1Features. Build from . If you will not put a prior on the intercept, centre and first (this is what scikit-learn does) and add the intercept back at the end.
- 2Initialize (e.g. 1) and (e.g. ). Precompute the eigenvalues of (or an SVD of ) once.
- 3Posterior. , , . (With the SVD, this is cheap for every new .)
- 4Re-estimate , , .
- 5Repeat steps 3–4 until stops changing (typically a handful of iterations). Skip 2–4 entirely if you have fixed from domain knowledge.
- 6Predict. For each new : mean , variance ; report e.g. mean sd.
Complexity. Forming costs and the inverse or eigendecomposition — the same as OLS. With the eigenbasis cached, each evidence iteration is only . Prediction is for the mean and for the variance. Memory is for , independent of , and sequential updating never needs the old rows.
Variants and practical tuning
The family
- BayesianRidge (sklearn)
- one shared prior precision for all weights, learned by evidence maximization. Posterior mean ≈ ridge with a data-chosen .
- ARD (ARDRegression)
- automatic relevance determination: one per weight. Irrelevant features get , so their weights are pruned to 0 — a Bayesian route to sparsity, like lasso.
- Unknown noise, conjugate
- a Normal–Inverse-Gamma prior on integrates the noise out analytically; the predictive becomes a Student- with heavier tails for small .
- Basis functions
- replace with polynomial or RBF features (see Polynomial regression); all formulas are unchanged, and the band now follows the curve.
- Gaussian processes
- the kernelized limit: put the prior directly on functions. Bayesian linear regression with basis functions is a GP with kernel .
- Non-conjugate models
- robust (Student-t) noise, hierarchical priors, Bayesian logistic regression: no closed form — use Laplace approximation, variational inference or MCMC (PyMC, Stan, NumPyro).
| Parameter | Default | Meaning / when to change |
|---|---|---|
alpha_1, alpha_2 | 1e-6 | Gamma hyperprior (shape, rate) on the noise precision; raise to encode a belief about the noise level |
lambda_1, lambda_2 | 1e-6 | Gamma hyperprior on the weight precision; raise lambda_1/lambda_2 to force more shrinkage |
alpha_init, lambda_init | 1/Var(y), 1 | starting values of the evidence iterations |
max_iter, tol | 300, 1e-3 | stop when the coefficients change by less than tol |
fit_intercept | True | centres X and y; the intercept gets no prior and no uncertainty |
compute_score | False | store the log marginal likelihood at every iteration in scores_ |
In code
1import numpy as np23def posterior(Phi, y, alpha, beta):4 """Gaussian posterior N(m, S) over weights; prior N(0, alpha^-1 I), noise precision beta."""5 A = alpha * np.eye(Phi.shape[1]) + beta * Phi.T @ Phi # posterior precision6 S = np.linalg.inv(A)7 m = beta * S @ Phi.T @ y8 return m, S910def predict(Phi_new, m, S, beta):11 mean = Phi_new @ m12 var = 1 / beta + np.einsum("ij,jk,ik->i", Phi_new, S, Phi_new)13 return mean, np.sqrt(var)1415def log_evidence(Phi, y, alpha, beta):16 N, M = Phi.shape17 m, S = posterior(Phi, y, alpha, beta)18 E = beta / 2 * np.sum((y - Phi @ m) ** 2) + alpha / 2 * m @ m19 return (M / 2 * np.log(alpha) + N / 2 * np.log(beta) - E20 - 0.5 * np.linalg.slogdet(np.linalg.inv(S))[1] - N / 2 * np.log(2 * np.pi))2122def fit_evidence(X, y, iters=300, tol=1e-3):23 """Empirical Bayes (MacKay) on centred data, like sklearn's BayesianRidge."""24 xm, ym = X.mean(0), y.mean()25 Xc, yc = X - xm, y - ym26 N = len(y)27 eig = np.linalg.eigvalsh(Xc.T @ Xc)28 alpha, beta = 1.0, 1 / (np.var(y) + 1e-12) # weight precision, noise precision29 w = np.zeros(X.shape[1])30 for _ in range(iters):31 m, S = posterior(Xc, yc, alpha, beta)32 gamma = np.sum(beta * eig / (alpha + beta * eig)) # effective no. of parameters33 alpha = (gamma + 2e-6) / (m @ m + 2e-6) # tiny Gamma hyperpriors, as sklearn34 beta = (N - gamma + 2e-6) / (np.sum((yc - Xc @ m) ** 2) + 2e-6)35 if np.abs(m - w).sum() < tol:36 break37 w = m38 m, S = posterior(Xc, yc, alpha, beta)39 return m, ym - xm @ m, S, alpha, beta, gamma, xm4041x = np.array([0.5, 1.0, 1.5, 2.5, 3.0, 4.0])42y = np.array([1.2, 2.0, 2.1, 3.1, 3.2, 4.4])43Phi = np.c_[np.ones_like(x), x]4445m, S = posterior(Phi, y, alpha=1.0, beta=4.0)46print("posterior mean", m.round(3), "sd", np.sqrt(np.diag(S)).round(3))47mean, sd = predict(np.array([[1, 2.0], [1, 6.0]]), m, S, beta=4.0)48print("predict x=2, 6:", mean.round(3), "+/-", sd.round(3))49print("log evidence (alpha=1, beta=4):", round(log_evidence(Phi, y, 1.0, 4.0), 3))5051w, b, S, alpha, beta, gamma, xm = fit_evidence(x[:, None], y)52print(f"evidence fit: w={w[0]:.4f} b={b:.4f} alpha={alpha:.4f} beta={beta:.4f} gamma={gamma:.4f}")53Xn = np.array([[2.0], [6.0]])54print("predict:", (Xn @ w + b).round(3), "+/-", np.sqrt(1 / beta + np.einsum("ij,jk,ik->i", Xn, S, Xn)).round(3))55Xc = Xn - xm # honest band: intercept uncertainty 1/(N beta), leverage measured from the data mean56print("centred band:", np.sqrt(1 / beta + 1 / (len(y) * beta) + np.einsum("ij,jk,ik->i", Xc, S, Xc)).round(3))5758# posterior mean [0.82 0.87] sd [0.374 0.158]59# predict x=2, 6: [2.56 6.04] +/- [0.539 0.831]60# log evidence (alpha=1, beta=4): -5.87161# evidence fit: w=0.8415 b=0.9134 alpha=1.4047 beta=30.9351 gamma=0.994862# predict: [2.597 5.963] +/- [0.217 0.407]63# centred band: [0.194 0.307]1import numpy as np2from sklearn.linear_model import BayesianRidge, LinearRegression34x = np.array([0.5, 1.0, 1.5, 2.5, 3.0, 4.0]).reshape(-1, 1)5y = np.array([1.2, 2.0, 2.1, 3.1, 3.2, 4.4])67model = BayesianRidge(compute_score=True).fit(x, y)8print(f"slope {model.coef_[0]:.4f} intercept {model.intercept_:.4f}")9print(f"noise precision alpha_ {model.alpha_:.4f} (noise sd {model.alpha_ ** -0.5:.4f})")10print(f"weight precision lambda_ {model.lambda_:.4f}")11print(f"log evidence {model.scores_[-1]:.4f} after {len(model.scores_) - 1} iterations")1213mean, std = model.predict(np.array([[2.0], [6.0]]), return_std=True)14print("predict x=2, 6:", mean.round(3), "+/-", std.round(3))15print("OLS slope", LinearRegression().fit(x, y).coef_.round(4))1617# slope 0.8415 intercept 0.913418# noise precision alpha_ 30.9351 (noise sd 0.1798)19# weight precision lambda_ 1.404720# log evidence -0.8487 after 4 iterations21# predict x=2, 6: [2.597 5.963] +/- [0.217 0.407]22# OLS slope [0.8459]The evidence decides the noise sd is about 0.18, not the 0.5 we assumed — the six points sit closer to a line than our guess — and : the data fully determine the one slope. The learned weight precision 1.40 shrinks the slope only slightly, from OLS's 0.8459 to 0.8415. With the tighter noise, the 6.0 forecast is . (scikit-learn's scores_ include the hyperprior terms, so they are not on the same scale as our −5.871.)
Use when
- +You need calibrated error bars per prediction — small data, extrapolation, decisions that depend on risk.
- +Data arrive one at a time and you want cheap online updates without storing the history.
- +You want ridge-like regularization but have too few rows to cross-validate reliably — maximize the evidence instead.
- +You have genuine prior knowledge about plausible coefficient sizes and want to encode it.
Avoid when
- −The relationship is strongly non-linear and you have no good basis functions — use a GP, trees or boosting.
- −Noise is heavy-tailed or has outliers: Gaussian likelihood lets one bad point drag the posterior (use Student-t noise via MCMC).
- − is huge (tens of thousands of features): the covariance is expensive; use variational or diagonal approximations.
- −You only need a point prediction on a big dataset — the posterior collapses to OLS/ridge and the extra machinery buys little.
Where it's used
Bayesian optimization
Surrogate models (Bayesian linear regression on learned features, or its GP limit) whose uncertainty drives which hyperparameters or experiments to try next.
Marketing mix modelling
Estimating the sales effect of each ad channel from a few years of weekly data, with priors from past campaigns and credible intervals for budget decisions.
Calibration curves in science
Instrument and dose–response calibration where every reading needs an uncertainty attached, and new measurements update the fit online.
Contextual bandits
Thompson sampling with a Bayesian linear model per action: sample weights from the posterior, act greedily on the sample — exploration comes from the uncertainty.
Check yourself
With prior precision and noise precision , the posterior mean equals ridge regression with which ?
Check yourself
Keep and collect more and more data across the same -range. The predictive sd at tends to…
Check yourself
Using the worked example's and , the predictive variance at is…
Check yourself
In scikit-learn's BayesianRidge, the fitted attribute alpha_ is…
Remember
- ✓Gaussian prior × Gaussian likelihood → Gaussian posterior with and .
- ✓The posterior mean is ridge regression with ; a flat prior () gives OLS.
- ✓Predictive variance : a constant noise floor plus weight uncertainty that grows away from the data and shrinks like .
- ✓Updating is sequential: each point adds βφφᵀ to the precision, and the posterior becomes the next prior.
- ✓Choose by maximizing the evidence (MacKay updates with , the effective number of parameters) — this is what BayesianRidge does.
- ✓The error bars are only trustworthy if the model is right and features are centred and scaled; check calibration on held-out data.