# Bayesian regression

> Put a prior on the weights and return a posterior distribution, so every prediction comes with its own uncertainty.

Source: https://www.mlfoundry.in/learn/bayesian-regression  
Section: ML Algorithms › Supervised · Regression  
Level: advanced · 17 min read · updated 2026-10-07

**Bayesian linear regression:** treats the weights $w$ of a linear model as **random variables**. It starts from a prior belief $p(w)$, multiplies by the likelihood of the data, and returns the **posterior** $p(w\mid\text{data})$ — a whole distribution over lines. Predictions average over that distribution, so each one comes with a mean *and* a variance that grows where the data are thin.

- **Task:** Regression (real-valued target)
- **Input → output:** features $x$ → a Gaussian predictive distribution $\mathcal N(\mu(x),\sigma^2(x))$, not just a number
- **Learns:** posterior mean $m$ and covariance $S$ of the weights; optionally the prior precision $\alpha$ and noise precision $\beta$
- **Training:** $O(NM^2+M^3)$ for $N$ rows and $M$ features (one $M\times M$ inverse); evidence maximization repeats a cheap update a few times
- **Prediction:** mean $O(M)$, variance $O(M^2)$ per point
- **Hyperparameters:** prior precision $\alpha$, noise precision $\beta$ (or the Gamma hyperpriors on them), the basis functions $\phi(x)$
- **Assumes:** linear in the features, Gaussian noise with constant variance, Gaussian prior on the weights

## The problem

Ordinary least squares (see **Linear regression** and **Multiple linear regression**) returns *one* line: the single $\hat w$ that minimizes squared error. It says nothing about how sure it is. Ask it to predict at $x=2$, right in the middle of the training data, and at $x=6$, far beyond it, and it answers both with the same confidence. With six points, those two answers deserve very different trust.

OLS has a second problem with small data: it overfits. Ridge regression fixes that with a penalty $\lambda\lVert w\rVert^2$, but $\lambda$ looks arbitrary and still has to be tuned by cross-validation, which is noisy when $N$ is tiny. Bayesian regression explains where the penalty comes from (a prior), gives error bars for free, and can pick $\lambda$ from the training data alone by maximizing the **evidence**.

## The idea

Before seeing data, say which lines are plausible: slopes and intercepts near zero, spread out with variance $1/\alpha$. Each data point then *rules out* lines that pass far from it — in weight space, a point $(x_n,y_n)$ is compatible only with the strip of $(w_0,w_1)$ where $w_0+w_1x_n\approx y_n$. Multiply the prior by all those strips and what survives is the posterior: an ellipse of lines that fit the data and are not too extreme. To predict, ask *every* line in the ellipse and report the average and the spread of their answers.

**Vocabulary**

- **Prior:** $p(w)=\mathcal N(0,\alpha^{-1}I)$: belief about the weights before seeing data. Large $\alpha$ = confident that weights are small.
- **Likelihood:** $p(y\mid X,w)$: how probable the observed targets are if the weights were $w$. Gaussian noise makes it $\prod_n\mathcal N(y_n\mid w^\top\phi_n,\beta^{-1})$.
- **Posterior:** $p(w\mid X,y)\propto$ likelihood × prior. For this model it is again Gaussian, $\mathcal N(m,S)$.
- **Precision:** inverse variance. $\alpha$ = prior precision of the weights; $\beta$ = precision of the noise ($\sigma^2=1/\beta$).
- **Conjugate prior:** a prior whose posterior has the same family. Gaussian prior + Gaussian likelihood → Gaussian posterior, so everything has a closed form.
- **Posterior predictive:** $p(y^*\mid x^*,\text{data})=\int p(y^*\mid x^*,w)\,p(w\mid\text{data})\,dw$: the forecast averaged over all plausible weights.
- **MAP estimate:** the single most probable $w$ (the posterior mode). Here it equals the posterior mean, and equals ridge regression.
- **Evidence (marginal likelihood):** $p(y\mid X,\alpha,\beta)$: probability of the data with the weights integrated out. Used to choose $\alpha,\beta$ — "empirical Bayes" or "type-II maximum likelihood".
- **Epistemic vs aleatoric:** uncertainty about $w$ (shrinks with more data) vs irreducible noise $1/\beta$ (does not).

## The math

Stack the inputs into a design matrix $\Phi$ whose row $n$ is $\phi(x_n)^\top$ — for a straight line, $\phi(x)=(1,x)$; for polynomial or RBF features, any fixed basis. The model and prior are:

$$
y_n = w^\top\phi(x_n)+\varepsilon_n,\quad \varepsilon_n\sim\mathcal N(0,\beta^{-1}),\qquad p(w)=\mathcal N(w\mid 0,\ \alpha^{-1}I)
$$

- $w\in\mathbb R^M$: weight vector (including the intercept, if φ has a constant 1)
- $\phi(x_n)$: feature vector of row n; Φ is the N×M matrix of all of them
- $\beta$: noise precision: noise variance is 1/β
- $\alpha$: prior precision: each weight has prior variance 1/α

**Derivation of the posterior.** Take logs of likelihood × prior and drop everything that does not depend on $w$:

$$
\ln p(w\mid y)= -\frac{\beta}{2}\lVert y-\Phi w\rVert^2-\frac{\alpha}{2}w^\top w+\text{const} = -\frac12 w^\top\!\big(\alpha I+\beta\Phi^\top\Phi\big)w+\beta\,w^\top\Phi^\top y+\text{const}
$$

- $\lVert y-\Phi w\rVert^2$: sum of squared residuals — the OLS loss
- $w^\top w$: squared L2 norm — the ridge penalty

This is a quadratic in $w$, so the posterior is Gaussian. A Gaussian $\mathcal N(m,S)$ has log-density $-\tfrac12 w^\top S^{-1}w+w^\top S^{-1}m+\text{const}$. Matching the quadratic term gives the precision; matching the linear term gives the mean:

$$
S^{-1}=A=\alpha I+\beta\,\Phi^\top\Phi,\qquad m=\beta\,S\,\Phi^\top y
$$

- $A$: posterior precision: prior precision plus β times the data's information Φᵀ Φ
- $S$: posterior covariance of the weights
- $m$: posterior mean (= mode = MAP estimate)

Divide inside by $\beta$: $m=(\Phi^\top\Phi+\tfrac{\alpha}{\beta}I)^{-1}\Phi^\top y$. That is exactly **ridge regression with $\lambda=\alpha/\beta$** — the ridge penalty is a Gaussian prior, and its strength is the ratio *prior precision / noise precision*. As $\alpha\to0$ (a flat prior) the mean becomes OLS.

**The predictive distribution.** A new target is $y^*=w^\top\phi^*+\varepsilon$ with $\phi^*=\phi(x^*)$. Under the posterior, $w^\top\phi^*$ is a linear function of a Gaussian, so it is Gaussian with mean $m^\top\phi^*$ and variance $\phi^{*\top}S\phi^*$; the noise $\varepsilon$ is independent, so the variances add:

$$
p(y^*\mid x^*,\text{data})=\mathcal N\!\Big(y^*\ \Big|\ m^\top\phi^*,\ \ \sigma^2(x^*)=\underbrace{\tfrac{1}{\beta}}_{\text{noise}}+\underbrace{\phi^{*\top}S\,\phi^*}_{\text{weight uncertainty}}\Big)
$$

- $1/\beta$: aleatoric term: the noise floor, the same everywhere
- $\phi^{*\top}S\phi^*$: epistemic term: large where x* is far from the training inputs, shrinks like 1/N

**Sequential learning.** Because $A=\alpha I+\beta\sum_n\phi_n\phi_n^\top$ is a sum over points, adding one observation just adds $\beta\phi_n\phi_n^\top$ to the precision: yesterday's posterior is today's prior. The order of the points does not matter, and you can update online without storing old data.

**Choosing α and β from the data: the evidence.** Integrating $w$ out of likelihood × prior (a Gaussian integral) gives the log marginal likelihood:

$$
\ln p(y\mid\alpha,\beta)=\frac M2\ln\alpha+\frac N2\ln\beta-E(m)-\frac12\ln|A|-\frac N2\ln 2\pi,\qquad E(m)=\frac\beta2\lVert y-\Phi m\rVert^2+\frac\alpha2 m^\top m
$$

- $E(m)$: data misfit plus weight size at the posterior mean (the ridge objective)
- $\tfrac12\ln|A|$: Occam penalty: grows when the data pin the weights down far more tightly than the prior did

Set the derivatives to zero. With $\lambda_i$ the eigenvalues of $\beta\Phi^\top\Phi$, $\partial/\partial\alpha$ gives $\tfrac{M}{2\alpha}-\tfrac12m^\top m-\tfrac12\sum_i\tfrac1{\alpha+\lambda_i}=0$; rearranging gives MacKay's fixed-point updates:

$$
\gamma=\sum_{i=1}^{M}\frac{\lambda_i}{\alpha+\lambda_i},\qquad \alpha\leftarrow\frac{\gamma}{m^\top m},\qquad \beta\leftarrow\frac{N-\gamma}{\lVert y-\Phi m\rVert^2}
$$

- $\gamma$: effective number of parameters, between 0 and M: how many weight directions the data actually determine
- $N-\gamma$: degrees of freedom left for estimating the noise (like N − p in the unbiased variance)

Alternate: compute $m,S$ with the current $\alpha,\beta$, update $\alpha,\beta$, repeat. This is what scikit-learn's `BayesianRidge` does (it also adds tiny Gamma hyperpriors, $10^{-6}$, so nothing divides by zero).

**What each hyperparameter does**

- **α ↑ (stronger prior):** adds $\alpha I$ to $A$: shrinks $m$ toward 0 and *narrows* the band. Too large and the model is confidently wrong (underfit).
- **α → 0:** flat prior: $m\to$ OLS, and $S\to(\beta\Phi^\top\Phi)^{-1}$, the classical OLS covariance.
- **β ↑ (less noise):** trusts each point more: the likelihood strips get thinner, the posterior tightens, and the noise floor $1/\beta$ drops.
- **α/β:** the only thing the posterior *mean* depends on — the ridge $\lambda$. The *band* depends on both separately.

## Worked example: six weeks of ad spend

A small shop records weekly ad spend $x$ (hundreds of dollars) and sales $y$ (thousands of units). We fix the prior precision $\alpha=1$ (weights have prior sd 1) and the noise precision $\beta=4$ (noise sd 0.5), use $\phi(x)=(1,x)$, and want forecasts at $x^*=2$ (inside the data) and $x^*=6$ (an extrapolation).

Six observations. $\sum x=12.5$, $\sum x^2=34.75$, $\sum y=16.0$, $\sum xy=40.7$.

| Week | x (ad spend) | y (sales) |
| --- | --- | --- |
| 1 | 0.5 | 1.2 |
| 2 | 1.0 | 2.0 |
| 3 | 1.5 | 2.1 |
| 4 | 2.5 | 3.1 |
| 5 | 3.0 | 3.2 |
| 6 | 4.0 | 4.4 |

**The posterior and two forecasts, by hand**

1. **Data information.** $\Phi^\top\Phi=\begin{pmatrix}N&\sum x\\\sum x&\sum x^2\end{pmatrix}=\begin{pmatrix}6&12.5\\12.5&34.75\end{pmatrix}$ and $\Phi^\top y=\begin{pmatrix}\sum y\\\sum xy\end{pmatrix}=\begin{pmatrix}16.0\\40.7\end{pmatrix}$.
2. **Posterior precision.** $A=\alpha I+\beta\Phi^\top\Phi=\begin{pmatrix}1+24&50\\50&1+139\end{pmatrix}=\begin{pmatrix}25&50\\50&140\end{pmatrix}$, with $|A|=25\cdot140-50^2=1000$.
3. **Posterior covariance.** $S=A^{-1}=\tfrac1{1000}\begin{pmatrix}140&-50\\-50&25\end{pmatrix}=\begin{pmatrix}0.14&-0.05\\-0.05&0.025\end{pmatrix}$.
4. **Posterior mean.** $\beta\Phi^\top y=(64,\ 162.8)$, so $m=S\,(64,162.8)^\top=(0.14\cdot64-0.05\cdot162.8,\ -0.05\cdot64+0.025\cdot162.8)=(\mathbf{0.82},\ \mathbf{0.87})$.
5. **Uncertainty in the weights.** $\operatorname{sd}(w_0)=\sqrt{0.14}=0.374$, $\operatorname{sd}(w_1)=\sqrt{0.025}=0.158$, correlation $-0.05/\sqrt{0.14\cdot0.025}=-0.845$: a steeper slope must come with a lower intercept to pass through the same points.
6. **Forecast at $x^*=2$.** Mean $0.82+0.87\cdot2=2.56$. Weight term $\phi^\top S\phi=0.14+2\cdot2\cdot(-0.05)+4\cdot0.025=0.04$. Variance $0.25+0.04=0.29$, sd $=0.539$ — barely above the noise floor of 0.5.
7. **Forecast at $x^*=6$.** Mean $0.82+0.87\cdot6=6.04$. Weight term $0.14-0.6+0.9=0.44$. Variance $0.25+0.44=0.69$, sd $=\mathbf{0.831}$. The 95% predictive interval is $6.04\pm1.96\cdot0.831=[4.41,\ 7.67]$.
8. **Compare.** OLS gives $(0.904,\ 0.846)$; ridge with $\lambda=\alpha/\beta=0.25$ (penalizing the intercept too) gives exactly $(0.82,\ 0.87)$ — the posterior mean. Bayes adds the band.

Now feed the same six points in one at a time, starting from the prior. Each row is the posterior after $n$ points — notice how the slope's sd and the forecast sd at $x^*=6$ collapse, but never below the noise floor $\sqrt{1/\beta}=0.5$.

Sequential updating with $\alpha=1,\ \beta=4$. Row $n=6$ is the batch answer above.

| n seen | mean (w₀, w₁) | sd(w₀) | sd(w₁) | forecast sd at x* = 6 |
| --- | --- | --- | --- | --- |
| 0 (prior) | (0, 0) | 1.000 | 1.000 | 6.103 |
| 1 | (0.800, 0.400) | 0.577 | 0.913 | 5.156 |
| 2 | (0.800, 0.933) | 0.577 | 0.707 | 3.819 |
| 3 | (0.824, 0.875) | 0.542 | 0.505 | 2.626 |
| 4 | (0.796, 0.912) | 0.452 | 0.295 | 1.493 |
| 5 | (0.875, 0.825) | 0.416 | 0.218 | 1.102 |
| 6 | **(0.820, 0.870)** | **0.374** | **0.158** | **0.831** |

> Figure: Left: weight space — the dashed prior circles (α = 1), the posterior 1σ and 2σ ellipses, the newest point's likelihood strip (orange) and the OLS estimate (×). Right: the same posterior as lines — mean, ±1σ/±2σ predictive band and five posterior samples (each coloured dot on the left is one coloured line on the right). Press play to add the six points one by one, or drag n, α, β or any point. At the defaults the readout matches the worked example: mean (0.820, 0.870), forecast at x* = 6 of 6.04 ± 0.831, ridge λ = 0.25.

> **Why the band is a bow-tie:** The weight term $\phi^\top S\phi=S_{00}+2xS_{01}+x^2S_{11}$ is a parabola in $x$ with its minimum near the centre of the training inputs. There, slope errors cancel; far away they are multiplied by the distance. So the band is narrowest at $x=-S_{01}/S_{11}=2.0$, next to the data mean $\bar x=2.08$, and opens up on both sides — at $x^*=6$ the weight term (0.44) is eleven times the one at $x^*=2$ (0.04).

Effect of the prior precision α on the same data (β = 4). The log evidence peaks near α ≈ 1.28 (value −5.845), so α = 1 is close to what the data prefer; α = 100 squashes the line toward zero *and* makes the band narrower — confidently wrong.

| α | mean (w₀, w₁) | forecast sd at x* = 6 | log evidence |
| --- | --- | --- | --- |
| 0.01 | (0.903, 0.846) | 0.855 | −9.656 |
| 0.1 | (0.895, 0.849) | 0.853 | −7.431 |
| 1 | (0.820, 0.870) | 0.831 | **−5.871** |
| 4 | (0.673, 0.903) | 0.785 | −6.695 |
| 10 | (0.544, 0.910) | 0.742 | −9.610 |
| 100 | (0.264, 0.626) | 0.633 | −40.579 |

## The algorithm, step by step

**Bayesian linear regression with evidence maximization**

1. **Features.** Build $\Phi$ from $\phi(x)$. If you will not put a prior on the intercept, centre $X$ and $y$ first (this is what scikit-learn does) and add the intercept back at the end.
2. **Initialize** $\alpha$ (e.g. 1) and $\beta$ (e.g. $1/\operatorname{Var}(y)$). Precompute the eigenvalues of $\Phi^\top\Phi$ (or an SVD of $\Phi$) once.
3. **Posterior.** $A=\alpha I+\beta\Phi^\top\Phi$, $S=A^{-1}$, $m=\beta S\Phi^\top y$. (With the SVD, this is cheap for every new $\alpha,\beta$.)
4. **Re-estimate** $\gamma=\sum_i\beta e_i/(\alpha+\beta e_i)$, $\alpha=\gamma/m^\top m$, $\beta=(N-\gamma)/\lVert y-\Phi m\rVert^2$.
5. **Repeat** steps 3–4 until $m$ stops changing (typically a handful of iterations). Skip 2–4 entirely if you have fixed $\alpha,\beta$ from domain knowledge.
6. **Predict.** For each new $x^*$: mean $m^\top\phi^*$, variance $1/\beta+\phi^{*\top}S\phi^*$; report e.g. mean $\pm1.96$ sd.

**Complexity.** Forming $\Phi^\top\Phi$ costs $O(NM^2)$ and the inverse or eigendecomposition $O(M^3)$ — the same as OLS. With the eigenbasis cached, each evidence iteration is only $O(M^2)$. Prediction is $O(M)$ for the mean and $O(M^2)$ for the variance. Memory is $O(M^2)$ for $S$, independent of $N$, and sequential updating never needs the old rows.

## Variants and practical tuning

**The family**

- **BayesianRidge (sklearn):** one shared prior precision for all weights, learned by evidence maximization. Posterior mean ≈ ridge with a data-chosen $\lambda$.
- **ARD (ARDRegression):** *automatic relevance determination*: one $\alpha_j$ per weight. Irrelevant features get $\alpha_j\to\infty$, so their weights are pruned to 0 — a Bayesian route to sparsity, like lasso.
- **Unknown noise, conjugate:** a Normal–Inverse-Gamma prior on $(w,\sigma^2)$ integrates the noise out analytically; the predictive becomes a Student-$t$ with heavier tails for small $N$.
- **Basis functions:** replace $(1,x)$ with polynomial or RBF features (see **Polynomial regression**); all formulas are unchanged, and the band now follows the curve.
- **Gaussian processes:** the kernelized limit: put the prior directly on functions. Bayesian linear regression with $M$ basis functions is a GP with kernel $k(x,x')=\alpha^{-1}\phi(x)^\top\phi(x')$.
- **Non-conjugate models:** robust (Student-t) noise, hierarchical priors, Bayesian logistic regression: no closed form — use Laplace approximation, variational inference or MCMC (PyMC, Stan, NumPyro).

scikit-learn BayesianRidge knobs. Note the naming: sklearn's alpha is the noise precision (our β) and its lambda is the weight precision (our α).

| Parameter | Default | Meaning / when to change |
| --- | --- | --- |
| `alpha_1`, `alpha_2` | 1e-6 | Gamma hyperprior (shape, rate) on the noise precision; raise to encode a belief about the noise level |
| `lambda_1`, `lambda_2` | 1e-6 | Gamma hyperprior on the weight precision; raise `lambda_1/lambda_2` to force more shrinkage |
| `alpha_init`, `lambda_init` | 1/Var(y), 1 | starting values of the evidence iterations |
| `max_iter`, `tol` | 300, 1e-3 | stop when the coefficients change by less than `tol` |
| `fit_intercept` | True | centres X and y; the intercept gets no prior and no uncertainty |
| `compute_score` | False | store the log marginal likelihood at every iteration in `scores_` |

> **Scale your features:** A single $\alpha$ applies the same prior variance to every weight, so a feature measured in millimetres and one in kilometres are shrunk very differently. Standardize features first (as you would for ridge), or use ARD so each weight gets its own precision.

## In code

```python
import numpy as np

def posterior(Phi, y, alpha, beta):
    """Gaussian posterior N(m, S) over weights; prior N(0, alpha^-1 I), noise precision beta."""
    A = alpha * np.eye(Phi.shape[1]) + beta * Phi.T @ Phi     # posterior precision
    S = np.linalg.inv(A)
    m = beta * S @ Phi.T @ y
    return m, S

def predict(Phi_new, m, S, beta):
    mean = Phi_new @ m
    var = 1 / beta + np.einsum("ij,jk,ik->i", Phi_new, S, Phi_new)
    return mean, np.sqrt(var)

def log_evidence(Phi, y, alpha, beta):
    N, M = Phi.shape
    m, S = posterior(Phi, y, alpha, beta)
    E = beta / 2 * np.sum((y - Phi @ m) ** 2) + alpha / 2 * m @ m
    return (M / 2 * np.log(alpha) + N / 2 * np.log(beta) - E
            - 0.5 * np.linalg.slogdet(np.linalg.inv(S))[1] - N / 2 * np.log(2 * np.pi))

def fit_evidence(X, y, iters=300, tol=1e-3):
    """Empirical Bayes (MacKay) on centred data, like sklearn's BayesianRidge."""
    xm, ym = X.mean(0), y.mean()
    Xc, yc = X - xm, y - ym
    N = len(y)
    eig = np.linalg.eigvalsh(Xc.T @ Xc)
    alpha, beta = 1.0, 1 / (np.var(y) + 1e-12)              # weight precision, noise precision
    w = np.zeros(X.shape[1])
    for _ in range(iters):
        m, S = posterior(Xc, yc, alpha, beta)
        gamma = np.sum(beta * eig / (alpha + beta * eig))   # effective no. of parameters
        alpha = (gamma + 2e-6) / (m @ m + 2e-6)            # tiny Gamma hyperpriors, as sklearn
        beta = (N - gamma + 2e-6) / (np.sum((yc - Xc @ m) ** 2) + 2e-6)
        if np.abs(m - w).sum() < tol:
            break
        w = m
    m, S = posterior(Xc, yc, alpha, beta)
    return m, ym - xm @ m, S, alpha, beta, gamma, xm

x = np.array([0.5, 1.0, 1.5, 2.5, 3.0, 4.0])
y = np.array([1.2, 2.0, 2.1, 3.1, 3.2, 4.4])
Phi = np.c_[np.ones_like(x), x]

m, S = posterior(Phi, y, alpha=1.0, beta=4.0)
print("posterior mean", m.round(3), "sd", np.sqrt(np.diag(S)).round(3))
mean, sd = predict(np.array([[1, 2.0], [1, 6.0]]), m, S, beta=4.0)
print("predict x=2, 6:", mean.round(3), "+/-", sd.round(3))
print("log evidence (alpha=1, beta=4):", round(log_evidence(Phi, y, 1.0, 4.0), 3))

w, b, S, alpha, beta, gamma, xm = fit_evidence(x[:, None], y)
print(f"evidence fit: w={w[0]:.4f} b={b:.4f} alpha={alpha:.4f} beta={beta:.4f} gamma={gamma:.4f}")
Xn = np.array([[2.0], [6.0]])
print("predict:", (Xn @ w + b).round(3), "+/-", np.sqrt(1 / beta + np.einsum("ij,jk,ik->i", Xn, S, Xn)).round(3))
Xc = Xn - xm   # honest band: intercept uncertainty 1/(N beta), leverage measured from the data mean
print("centred band:", np.sqrt(1 / beta + 1 / (len(y) * beta) + np.einsum("ij,jk,ik->i", Xc, S, Xc)).round(3))

# posterior mean [0.82 0.87] sd [0.374 0.158]
# predict x=2, 6: [2.56 6.04] +/- [0.539 0.831]
# log evidence (alpha=1, beta=4): -5.871
# evidence fit: w=0.8415 b=0.9134 alpha=1.4047 beta=30.9351 gamma=0.9948
# predict: [2.597 5.963] +/- [0.217 0.407]
# centred band: [0.194 0.307]
```

From scratch with NumPy: fixed-α/β posterior (the worked example) and MacKay evidence maximization. Prints mean (0.82, 0.87), forecasts 2.56 ± 0.539 and 6.04 ± 0.831, then the evidence fit that matches scikit-learn below.

```python
import numpy as np
from sklearn.linear_model import BayesianRidge, LinearRegression

x = np.array([0.5, 1.0, 1.5, 2.5, 3.0, 4.0]).reshape(-1, 1)
y = np.array([1.2, 2.0, 2.1, 3.1, 3.2, 4.4])

model = BayesianRidge(compute_score=True).fit(x, y)
print(f"slope {model.coef_[0]:.4f}  intercept {model.intercept_:.4f}")
print(f"noise precision alpha_ {model.alpha_:.4f}  (noise sd {model.alpha_ ** -0.5:.4f})")
print(f"weight precision lambda_ {model.lambda_:.4f}")
print(f"log evidence {model.scores_[-1]:.4f}  after {len(model.scores_) - 1} iterations")

mean, std = model.predict(np.array([[2.0], [6.0]]), return_std=True)
print("predict x=2, 6:", mean.round(3), "+/-", std.round(3))
print("OLS slope", LinearRegression().fit(x, y).coef_.round(4))

# slope 0.8415  intercept 0.9134
# noise precision alpha_ 30.9351  (noise sd 0.1798)
# weight precision lambda_ 1.4047
# log evidence -0.8487  after 4 iterations
# predict x=2, 6: [2.597 5.963] +/- [0.217 0.407]
# OLS slope [0.8459]
```

scikit-learn's BayesianRidge on the worked example: α and β are learned by evidence maximization instead of being fixed.

The evidence decides the noise sd is about 0.18, not the 0.5 we assumed — the six points sit closer to a line than our guess — and $\gamma=0.995$: the data fully determine the one slope. The learned weight precision 1.40 shrinks the slope only slightly, from OLS's 0.8459 to 0.8415. With the tighter noise, the 6.0 forecast is $5.963\pm0.407$. (scikit-learn's `scores_` include the hyperprior terms, so they are not on the same scale as our −5.871.)

> **return_std depends on where zero is:** `BayesianRidge.predict(..., return_std=True)` computes the weight term as $x^\top S x$ on the **raw** features and ignores intercept uncertainty, so the band is narrowest at $x=0$, not at the data mean. Shift the same data by +10 and refit: the mean forecasts are identical, but the sd at the same two points jumps from 0.217 and 0.407 to 0.751 and 0.989. Standardize features (centre them) before fitting if you will use the error bars; on centred data sklearn gives 0.180 and 0.298, close to the honest 0.194 and 0.307 printed by the scratch code.

> **The band is only as honest as the model:** The predictive variance assumes the model is right: linear in $\phi$, Gaussian noise with one constant variance. If the truth is curved, the band around a straight line still shrinks like $1/N$ — the model grows *confident* in a wrong answer. Heteroscedastic noise (noise growing with $x$) is also invisible to it. Check residuals, and check calibration on held-out data: about 95% of targets should fall inside the 95% interval.

> **Two alphas:** Bishop and most textbooks write α for the weight precision and β for the noise precision. scikit-learn's `BayesianRidge` uses `alpha_` for the **noise** precision and `lambda_` for the **weight** precision. Read `alpha_` as 1/σ², not as a regularization strength.

**Strengths**

- You need calibrated error bars per prediction — small data, extrapolation, decisions that depend on risk.
- Data arrive one at a time and you want cheap online updates without storing the history.
- You want ridge-like regularization but have too few rows to cross-validate $\lambda$ reliably — maximize the evidence instead.
- You have genuine prior knowledge about plausible coefficient sizes and want to encode it.

**Limitations**

- The relationship is strongly non-linear and you have no good basis functions — use a GP, trees or boosting.
- Noise is heavy-tailed or has outliers: Gaussian likelihood lets one bad point drag the posterior (use Student-t noise via MCMC).
- $M$ is huge (tens of thousands of features): the $M\times M$ covariance is expensive; use variational or diagonal approximations.
- You only need a point prediction on a big dataset — the posterior collapses to OLS/ridge and the extra machinery buys little.

- **Bayesian optimization:** Surrogate models (Bayesian linear regression on learned features, or its GP limit) whose uncertainty drives which hyperparameters or experiments to try next.
- **Marketing mix modelling:** Estimating the sales effect of each ad channel from a few years of weekly data, with priors from past campaigns and credible intervals for budget decisions.
- **Calibration curves in science:** Instrument and dose–response calibration where every reading needs an uncertainty attached, and new measurements update the fit online.
- **Contextual bandits:** Thompson sampling with a Bayesian linear model per action: sample weights from the posterior, act greedily on the sample — exploration comes from the uncertainty.

**Question:** With prior precision $\alpha=1$ and noise precision $\beta=4$, the posterior mean equals ridge regression with which $\lambda$?

- A. 4
- B. 1
- C. 0.25
- D. 0.2

**Answer:** C. 0.25 — $m=(\Phi^\top\Phi+\tfrac{\alpha}{\beta}I)^{-1}\Phi^\top y$, so $\lambda=\alpha/\beta=1/4=0.25$. That is why the worked example's posterior mean (0.82, 0.87) matches ridge with λ = 0.25 exactly.

**Question:** Keep $\beta=4$ and collect more and more data across the same $x$-range. The predictive sd at $x^*=2$ tends to…

- A. 0
- B. 0.5
- C. 1
- D. it keeps growing

**Answer:** B. 0.5 — The weight term $\phi^\top S\phi$ shrinks like $1/N$, but the noise term $1/\beta=0.25$ does not, so the sd tends to $\sqrt{0.25}=0.5$. Epistemic uncertainty vanishes; aleatoric does not.

**Question:** Using the worked example's $S=\begin{pmatrix}0.14&-0.05\\-0.05&0.025\end{pmatrix}$ and $\beta=4$, the predictive variance at $x^*=0$ is…

- A. 0.14
- B. 0.25
- C. 0.39
- D. 0.69

**Answer:** C. 0.39 — $\phi=(1,0)$, so $\phi^\top S\phi=S_{00}=0.14$ (the intercept's variance), plus noise $0.25$: total $0.39$, sd $0.624$. 0.69 is the value at $x^*=6$.

**Question:** In scikit-learn's BayesianRidge, the fitted attribute `alpha_` is…

- A. the ridge penalty λ
- B. the precision of the weights' prior
- C. the precision of the noise, 1/σ²
- D. the effective number of parameters

**Answer:** C. the precision of the noise, 1/σ² — sklearn swaps the textbook names: `alpha_` is the noise precision (30.94 on the worked example, i.e. noise sd 0.18) and `lambda_` is the weight precision (1.40).

**Key points**

- Gaussian prior $\mathcal N(0,\alpha^{-1}I)$ × Gaussian likelihood → Gaussian posterior with $S^{-1}=\alpha I+\beta\Phi^\top\Phi$ and $m=\beta S\Phi^\top y$.
- The posterior mean is ridge regression with $\lambda=\alpha/\beta$; a flat prior ($\alpha\to0$) gives OLS.
- Predictive variance $=1/\beta+\phi^\top S\phi$: a constant noise floor plus weight uncertainty that grows away from the data and shrinks like $1/N$.
- Updating is sequential: each point adds βφφᵀ to the precision, and the posterior becomes the next prior.
- Choose $\alpha,\beta$ by maximizing the evidence (MacKay updates with $\gamma$, the effective number of parameters) — this is what BayesianRidge does.
- The error bars are only trustworthy if the model is right and features are centred and scaled; check calibration on held-out data.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
