Multiple linear regression

With several features, ŷ = Xw fits a hyperplane; each coefficient is the effect of its feature with the others held fixed.

16 min read1743 wordscore10 viewsupdated 2026-10-07

Definition

Multiple linear regression predicts a real-valued target from pp features at once, y^=b0+b1x1+⋯+bpxp\hat y = b_0 + b_1x_1 + \dots + b_px_p, fitting all coefficients jointly by least squares. Each bjb_j is the change in y^\hat y per unit of xjx_j with every other feature held fixed — not the slope you would get by regressing yy on xjx_j alone.

At a glance

Task
Regression — predict a real number from several features
Input → output
x∈Rpx\in\mathbb{R}^p → y^=b0+x⊤b\hat y = b_0 + x^\top b
Learns
p+1p+1 numbers: an intercept and one partial slope per feature
Training
form X⊤XX^\top X in O(np2)O(np^2), solve in O(p3)O(p^3) (QR or Cholesky); or gradient descent at O(np)O(np) per step
Prediction
O(p)O(p) — one dot product
Hyperparameters
none for plain OLS (just whether to fit an intercept); which features and interactions to include is the real modelling choice
Assumes
yy is linear in the (possibly transformed) features; errors independent, mean zero, constant variance; no feature is an exact linear combination of others

The problem

The linear regression article fits a line to one feature. Real targets depend on many things at once — a house price on size, age, location, condition — and those things are correlated with each other. Regressing the target on each feature separately then gives answers that are not just imprecise but wrong in sign.

In the eight houses of the worked example below, bigger houses also tend to be older (correlation 0.94). Regress price on size alone and each extra m² seems worth only 0.80 $k. Regress price on age alone and each extra year seems to add 1.35 $k. Fit both together and the picture changes: an extra m² is worth 2.00 $k at a fixed age, and each year costs 2.81 $k at a fixed size. The one-at-a-time slopes were each soaking up the other feature's effect.

The idea

Fit one model with all features in it. With two features the model is a plane over the (x1,x2)(x_1,x_2) floor; with pp features, a hyperplane in p+1p+1 dimensions. Least squares tilts it in every direction at once until the sum of squared vertical errors is smallest.

The tilt along x1x_1 is b1b_1. Moving along the x1x_1 direction of a plane means x2x_2 does not change — so b1b_1 answers "two houses of the same age, one 1 m² larger: how much more does it cost?". Equivalently (a theorem we derive below), b1b_1 is the slope you get after removing from both size and price whatever age can explain, then regressing what is left over. That is what "held fixed" means mechanically.

Vocabulary

Design matrix X
n×(p+1)n\times(p+1) table: one row per example, a column of 1s for the intercept, then one column per feature
Coefficient / partial slope
bjb_j: change in y^\hat y per unit of xjx_j with all other features unchanged
Intercept
b0b_0: prediction when every feature is 0 (often outside the data — e.g. a 0 m² house)
Fitted value, residual
y^i=xi⊤b\hat y_i = x_i^\top b and ei=yi−y^ie_i = y_i-\hat y_i
Hat matrix
H=X(X⊤X)−1X⊤H = X(X^\top X)^{-1}X^\top, the projection with y^=Hy\hat y = Hy
Confounder / omitted variable
a feature that affects y and is correlated with an included feature; leaving it out biases that feature's coefficient
Multicollinearity
features that are strongly linearly related to each other; the fit is fine but individual coefficients become unstable
VIF
variance inflation factor 1/(1−Rj2)1/(1-R_j^2): how much correlation with the other features inflates Var⁡(bj)\operatorname{Var}(b_j)
Dummy (indicator)
a 0/1 column encoding one level of a categorical feature
Interaction term
a product column xjxkx_jx_k letting the slope of xjx_j depend on xkx_k

The math

Stack the nn examples into a design matrix and write the whole model at once:

y=Xβ+ε,X=[1x11⋯x1p⋮⋮⋮1xn1⋯xnp]y = X\beta + \varepsilon,\qquad X=\begin{bmatrix}1 & x_{11} & \cdots & x_{1p}\\ \vdots & \vdots & & \vdots\\ 1 & x_{n1} & \cdots & x_{np}\end{bmatrix}
yy
nn-vector of targets
XX
n×(p+1)n\times(p+1) design matrix; row ii is xi⊤x_i^\top with a leading 1
β\beta
true coefficients (β0,…,βp)(\beta_0,\dots,\beta_p); our estimate is bb
ε\varepsilon
noise, mean 0, variance σ2\sigma^2, independent across rows
SSE(b)=∥y−Xb∥2=y⊤y−2b⊤X⊤y+b⊤X⊤X b\text{SSE}(b)=\lVert y-Xb\rVert^2 = y^\top y - 2b^\top X^\top y + b^\top X^\top X\, b
SSE\text{SSE}
sum of squared errors (n × the MSE); same minimiser

It is a quadratic bowl in p+1p+1 dimensions. Its gradient is ∇bSSE=−2X⊤(y−Xb)\nabla_b \text{SSE} = -2X^\top(y-Xb); setting it to zero gives the normal equations and, when X⊤XX^\top X is invertible, the closed form:

X⊤X b=X⊤y⟹b=(X⊤X)−1X⊤yX^\top X\,b = X^\top y \quad\Longrightarrow\quad b=(X^\top X)^{-1}X^\top y
X⊤XX^\top X
(p+1)×(p+1)(p+1)\times(p+1) Gram matrix: sums of products of feature pairs
X⊤yX^\top y
sums of products of each feature with the target

The normal equations say X⊤e=0X^\top e = 0: the residual vector is orthogonal to every column of XX. So the residuals sum to zero (the ones column) and are uncorrelated with every feature in the sample. Geometrically, y^=Hy\hat y=Hy is the orthogonal projection of yy onto the (p+1)(p+1)-dimensional subspace spanned by the columns. X⊤XX^\top X is invertible exactly when no column is a linear combination of the others; the Hessian 2X⊤X2X^\top X is then positive definite and the minimum is unique.

Two features by hand: center first

Subtracting the means removes the intercept from the system (it is recovered at the end), leaving a 2×22\times2 problem in centered sums Sjk=∑i(xij−xˉj)(xik−xˉk)S_{jk}=\sum_i (x_{ij}-\bar x_j)(x_{ik}-\bar x_k) and Sjy=∑i(xij−xˉj)(yi−yˉ)S_{jy}=\sum_i (x_{ij}-\bar x_j)(y_i-\bar y):

[S11S12S12S22][b1b2]=[S1yS2y]  ⇒  b1=S22S1y−S12S2yD,    b2=S11S2y−S12S1yD\begin{bmatrix}S_{11}&S_{12}\\S_{12}&S_{22}\end{bmatrix}\begin{bmatrix}b_1\\b_2\end{bmatrix}=\begin{bmatrix}S_{1y}\\S_{2y}\end{bmatrix}\;\Rightarrow\; b_1=\frac{S_{22}S_{1y}-S_{12}S_{2y}}{D},\;\; b_2=\frac{S_{11}S_{2y}-S_{12}S_{1y}}{D}
DD
determinant S11S22−S122S_{11}S_{22}-S_{12}^2; near 0 when the features are nearly collinear
b0b_0
recovered as yˉ−b1xˉ1−b2xˉ2\bar y - b_1\bar x_1 - b_2\bar x_2 (the plane passes through the means)

If the features are uncorrelated (S12=0S_{12}=0) this collapses to b1=S1y/S11b_1=S_{1y}/S_{11} — the simple-regression slope. Only correlation between features makes the joint fit differ from the one-at-a-time fits.

What "held fixed" means: Frisch–Waugh–Lovell

b1=∑ix~i1 y~i∑ix~i12,x~1=x1−x1∣x2^,y~=y−y∣x2^b_1=\frac{\sum_i \tilde x_{i1}\,\tilde y_i}{\sum_i \tilde x_{i1}^2},\qquad \tilde x_1 = x_1 - \widehat{x_1\mid x_2},\quad \tilde y = y-\widehat{y\mid x_2}
x~1\tilde x_1
residual of regressing x1x_1 on the other features: the part of size that age cannot predict
y~\tilde y
residual of regressing yy on the other features

Derivation. Write the fit as y=b1x1+Zc+ey = b_1x_1 + Z c + e where ZZ holds the intercept and every other feature, and let M=I−Z(Z⊤Z)−1Z⊤M = I - Z(Z^\top Z)^{-1}Z^\top be the "residual-maker" that removes anything ZZ explains (MZ=0MZ=0, M=M⊤=M2M=M^\top=M^2). Multiply by MM: My=b1Mx1+MeMy = b_1 Mx_1 + Me. Because ee is orthogonal to every column of XX, it is orthogonal to ZZ, so Me=eMe=e; then (Mx1)⊤e=x1⊤Me=x1⊤e=0(Mx_1)^\top e = x_1^\top Me = x_1^\top e = 0. Taking the dot product of My=b1Mx1+eMy = b_1 Mx_1 + e with Mx1Mx_1 leaves (Mx1)⊤My=b1 (Mx1)⊤Mx1(Mx_1)^\top My = b_1\,(Mx_1)^\top Mx_1, which is the formula with x~1=Mx1\tilde x_1 = Mx_1, y~=My\tilde y = My. A multiple-regression coefficient is a simple-regression slope on residualized data.

Omitted-variable bias

b1simple=b1+b2 δ,δ=S12S11b^{\text{simple}}_1 = b_1 + b_2\,\delta,\qquad \delta=\frac{S_{12}}{S_{11}}
b1simpleb^{\text{simple}}_1
slope from regressing yy on x1x_1 alone
δ\delta
slope from regressing the omitted x2x_2 on x1x_1

Substitute S1y=b1S11+b2S12S_{1y} = b_1S_{11}+b_2S_{12} (the first normal equation) into b1simple=S1y/S11b^{\text{simple}}_1 = S_{1y}/S_{11}. This is an exact identity on any sample. The bias b2δb_2\delta vanishes only if the omitted feature has no effect (b2=0b_2=0) or is uncorrelated with the included one (δ=0\delta=0). In the houses: δ=0.426\delta=0.426 years per m², so 0.80=2.00+(−2.81)(0.426)0.80 = 2.00 + (-2.81)(0.426).

How sure are we about each coefficient?

Var⁡(b)=σ2(X⊤X)−1,Var⁡(bj)=σ2Sjj (1−Rj2),σ^2=SSEn−p−1\operatorname{Var}(b)=\sigma^2(X^\top X)^{-1},\qquad \operatorname{Var}(b_j)=\frac{\sigma^2}{S_{jj}\,(1-R_j^2)},\qquad \hat\sigma^2=\frac{\text{SSE}}{n-p-1}
Rj2R_j^2
R2R^2 from regressing xjx_j on the other features
11−Rj2\tfrac{1}{1-R_j^2}
the VIF: the variance multiplier caused by collinearity
n−p−1n-p-1
residual degrees of freedom (p+1p+1 parameters were fitted)

The second form follows from FWL: bjb_j is a simple slope on x~j\tilde x_j, whose variance is σ2/∑x~ij2\sigma^2/\sum\tilde x_{ij}^2, and ∑x~ij2=Sjj(1−Rj2)\sum\tilde x_{ij}^2 = S_{jj}(1-R_j^2). So collinearity does not bias anything — it shrinks the leftover variation x~j\tilde x_j the coefficient can learn from, and the standard error grows by VIF\sqrt{\text{VIF}}. Here the two features correlate at r=0.94r=0.94, so VIF=1/(1−0.887)=8.86\text{VIF}=1/(1-0.887)=8.86 and each standard error is about 3× what it would be with uncorrelated features.

R2=1−SSESST,Radj2=1−(1−R2)n−1n−p−1R^2 = 1-\frac{\text{SSE}}{\text{SST}},\qquad R^2_{\text{adj}} = 1-(1-R^2)\frac{n-1}{n-p-1}
SST\text{SST}
∑(yi−yˉ)2\sum(y_i-\bar y)^2, the error of predicting the mean
Radj2R^2_{\text{adj}}
R² penalized for the number of features; it can fall when a useless feature is added

Adding a column can never increase the minimum SSE (the old solution, with the new coefficient at 0, is still available), so plain R2R^2 never falls as features are added — even pure noise features. Adjusted R2R^2, held-out error, or cross-validation are the honest comparisons.

Worked example: price from size and age

Eight houses. Size and age rise together (correlation 0.94), which is exactly the situation where one-feature-at-a-time thinking fails.

HouseSize x₁ (m²)Age x₂ (years)Price y ($k)
1505138
26012133
3708161
48020139
59018173
610030163
711026195
812037184
Eight houses. Means: size 85 m², age 19.5 years, price 160.75 $k.

Fitting the plane by hand

  1. 1Means. xˉ1=85\bar x_1 = 85, xˉ2=19.5\bar x_2 = 19.5, yˉ=160.75\bar y = 160.75.
  2. 2Centered sums. S11=4200S_{11}=4200, S22=860S_{22}=860, S12=1790S_{12}=1790, S1y=3360S_{1y}=3360, S2y=1159S_{2y}=1159 (and SST=Syy=3629.5\text{SST}=S_{yy}=3629.5).
  3. 3Determinant. D=4200⋅860−17902=3,612,000−3,204,100=407,900D = 4200\cdot860 - 1790^2 = 3{,}612{,}000-3{,}204{,}100 = 407{,}900.
  4. 4Size coefficient. b1=860⋅3360−1790⋅1159407,900=2,889,600−2,074,610407,900=814,990407,900=1.998b_1 = \dfrac{860\cdot3360 - 1790\cdot1159}{407{,}900} = \dfrac{2{,}889{,}600-2{,}074{,}610}{407{,}900}=\dfrac{814{,}990}{407{,}900} = 1.998 $k per m².
  5. 5Age coefficient. b2=4200⋅1159−1790⋅3360407,900=4,867,800−6,014,400407,900=−2.811b_2 = \dfrac{4200\cdot1159-1790\cdot3360}{407{,}900}=\dfrac{4{,}867{,}800-6{,}014{,}400}{407{,}900} = -2.811 $k per year.
  6. 6Intercept. b0=160.75−1.998(85)−(−2.811)(19.5)=160.75−169.83+54.81=45.73b_0 = 160.75 - 1.998(85) - (-2.811)(19.5) = 160.75-169.83+54.81 = 45.73.
  7. 7Fit quality. SSE=174.10\text{SSE}=174.10, so R2=1−174.10/3629.5=0.952R^2 = 1-174.10/3629.5 = 0.952 and Radj2=1−0.048⋅75=0.933R^2_{\text{adj}} = 1-0.048\cdot\tfrac{7}{5} = 0.933; σ^2=174.10/5=34.82\hat\sigma^2 = 174.10/5 = 34.82.
  8. 8Compare. Size alone: b1simple=S1y/S11=3360/4200=0.80b^{\text{simple}}_1 = S_{1y}/S_{11} = 3360/4200 = 0.80, R2=0.741R^2 = 0.741. Check the bias identity: δ=1790/4200=0.426\delta = 1790/4200 = 0.426 and 1.998+(−2.811)(0.426)=0.801.998 + (-2.811)(0.426) = 0.80. ✓
  9. 9Predict a new 85 m², 15-year-old house: y^=45.73+1.998(85)−2.811(15)=45.73+169.83−42.16=173.40\hat y = 45.73 + 1.998(85) - 2.811(15) = 45.73 + 169.83 - 42.16 = 173.40 k.Thesize−onlymodelsays160.75k. The size-only model says 160.75 k (85 m² is the mean size) — it cannot see that this house is 4.5 years younger than average, worth 4.5×2.811=12.654.5\times2.811=12.65 $k — exactly the gap between the two predictions.
Houseŷ ($k)Residual y − ŷSize residual x̃₁ (FWL)Price residual ỹ (FWL)
1131.586.42−4.82−3.21
2131.881.12−9.39−17.64
3163.11−2.118.9415.75
4149.35−10.35−6.04−22.42
5174.96−1.968.1214.27
6161.201.80−6.85−11.90
7192.432.5711.4725.49
8181.492.51−1.42−0.33
Fitted values and residuals of ŷ = 45.73 + 1.998·size − 2.811·age. The residuals sum to 0 and are orthogonal to both feature columns, as the normal equations require. House 4 (cheap for its size and age) has the largest residual.

The last two columns are FWL in action: regress size on age (slope 1790/860=2.0811790/860=2.081 m² per year) and price on age (slope 1159/860=1.3481159/860 = 1.348 kperyear),keeptheresiduals,andtheslopeofk per year), keep the residuals, and the slope of \tilde yon on \tilde x_1isagainexactly is again exactly 1.998.Withuncertainty:. With uncertainty: b_1 = 1.998 \pm 0.696and and b_2 = -2.811\pm1.539(95 (95% intervals, standard errors 0.271 and 0.599, twith5degreesoffreedom)—wide,becauseVIF with 5 degrees of freedom) — wide, because VIF = 8.86$.

colour = age: ● new → ● old · drag a house up/down5060708090100110120120140160180200size (m²)price ($k)slope 0.805y12y8y20y18y30y26y37y
ignore age: price = 92.75 + 0.80·size · R² = 0.741
with age: price = 45.73 + 2.00·size − 2.81·age · R² = 0.952
solve: b₁ = (S₂₂S₁y − S₁₂S₂y)/D = (860·3360 − 1790·1159)/407900 = 1.998
omitted age: 0.800 = b₁ 1.998 + b₂ (-2.811) × δ 0.426
the simple slope mixes size's own effect with that of age, which rises with size
fig.The eight houses, coloured by age. (1) Ignoring age, the best line has slope 0.80 (R² 0.741). (2) The joint fit is a plane; slicing it at ages 10, 20 and 30 gives parallel lines of slope 2.00, 28.1 $k apart per 10 years (R² 0.952). (3) Partialling out age moves every house to its FWL residual coordinates, and the line through them has slope exactly 2.00. Drag any house up or down in views 1–2: all fits, the solve and the omitted-variable identity update live.

The algorithm, step by step

Ordinary least squares with p features

  1. 1Build X. One row per example; prepend a column of 1s; one-hot encode categoricals, dropping one level per feature to avoid exact collinearity with the intercept.
  2. 2Check rank. If a column is a linear combination of others (duplicate unit, all dummies kept, a sum column), X⊤XX^\top X is singular — drop or merge columns.
  3. 3Solve. Compute a QR decomposition X=QRX = QR and back-substitute Rb=Q⊤yRb = Q^\top y (O(np2)O(np^2)). This avoids forming X⊤XX^\top X, which squares the condition number. (Cholesky on X⊤XX^\top X is faster but less stable; SVD handles rank deficiency.)
  4. 4Diagnose. Residuals e=y−Xbe = y - Xb; R2R^2, adjusted R2R^2; σ^2=∥e∥2/(n−p−1)\hat\sigma^2 = \lVert e\rVert^2/(n-p-1); standard errors from σ^2(X⊤X)−1\hat\sigma^2(X^\top X)^{-1}; VIFs; residual-vs-fitted plots for curvature and changing spread.
  5. 5Predict. y^=b0+x⊤b\hat y = b_0 + x^\top b — one dot product per new example.

Complexity. Training is O(np2+p3)O(np^2 + p^3) time and O(p2)O(p^2) extra memory for the Gram matrix; with nn in the millions and pp in the hundreds this is seconds. When pp reaches tens of thousands, or data arrives in a stream, switch to (stochastic) gradient descent on the same loss, b←b+2ηnX⊤(y−Xb)b \leftarrow b + \tfrac{2\eta}{n}X^\top(y - Xb), at O(np)O(np) per pass — standardize features first, or the bowl is long and thin and descent zig-zags.

Variants and practical choices

ChoiceWhat it doesWhen
Dummy variablesa 0/1 column per category level (minus one); its coefficient is the shift vs the dropped level, others fixedany categorical feature
Interaction x₁·x₂slope of x₁ becomes b₁ + b₁₂x₂the effect of one feature depends on another
Log transform of ycoefficients become approximate % changesmultiplicative effects, right-skewed targets (prices, incomes)
Standardize featurescoefficients in units of 1 SD; comparable magnitudes; better GD conditioningcomparing importance, gradient descent, before ridge/lasso
Ridge / lasso / elastic netpenalize coefficient size to stabilise or selectcollinearity, p close to n, many weak features
Robust (HC) standard errorsvalid intervals when error variance changes with xresidual spread fans out
Weighted least squaresminimise Σ wᵢ eᵢ²known unequal noise per row
Things you actually decide when fitting a multiple regression.

Standardized coefficients make the houses comparable: bj⋅sxj/syb_j\cdot s_{x_j}/s_y gives 2.152.15 for size and −1.37-1.37 for age — one SD of size moves price by more than one SD of age does, in either direction.

In code

python
1import numpy as np
2
3def fit_ols(X, y):
4 """Least squares with an intercept. Returns (w, b)."""
5 A = np.c_[np.ones(len(X)), X] # design matrix with a ones column
6 beta, *_ = np.linalg.lstsq(A, y, rcond=None) # QR/SVD: stable even if XᵀX is ill-conditioned
7 return beta[1:], beta[0]
8
9def summary(X, y, w, b):
10 n, p = X.shape
11 A = np.c_[np.ones(n), X]
12 r = y - (X @ w + b)
13 sse, sst = r @ r, ((y - y.mean()) ** 2).sum()
14 r2 = 1 - sse / sst
15 adj = 1 - (1 - r2) * (n - 1) / (n - p - 1)
16 s2 = sse / (n - p - 1) # unbiased noise variance
17 se = np.sqrt(np.diag(s2 * np.linalg.inv(A.T @ A)))
18 return r2, adj, se
19
20def vif(X):
21 out = []
22 for j in range(X.shape[1]):
23 others = np.delete(X, j, axis=1)
24 w, b = fit_ols(others, X[:, j])
25 r = X[:, j] - (others @ w + b)
26 r2 = 1 - (r @ r) / ((X[:, j] - X[:, j].mean()) ** 2).sum()
27 out.append(1 / (1 - r2))
28 return np.array(out)
29
30size = np.array([50, 60, 70, 80, 90, 100, 110, 120.])
31age = np.array([5, 12, 8, 20, 18, 30, 26, 37.])
32price = np.array([138, 133, 161, 139, 173, 163, 195, 184.])
33X = np.c_[size, age]
34
35w, b = fit_ols(X, price)
36r2, adj, se = summary(X, price, w, b)
37print("w =", w.round(3), " b =", round(b, 2)) # [ 1.998 -2.811] 45.73
38print("R2 =", round(r2, 3), " adj R2 =", round(adj, 3)) # 0.952 0.933
39print("SE (b, size, age) =", se.round(3)) # [12.828 0.271 0.599]
40print("VIF =", vif(X).round(2)) # [8.86 8.86]
41print("price(85 m2, 15 y) =", round(np.array([85, 15]) @ w + b, 2)) # 173.4
From scratch with NumPy: least squares, R², standard errors and VIF on the eight houses.
python
1import numpy as np, pandas as pd
2from scipy import stats
3from sklearn.linear_model import LinearRegression
4
5df = pd.DataFrame({
6 "size": [50, 60, 70, 80, 90, 100, 110, 120],
7 "age": [5, 12, 8, 20, 18, 30, 26, 37],
8 "price": [138, 133, 161, 139, 173, 163, 195, 184],
9})
10X, y = df[["size", "age"]], df.price
11
12both = LinearRegression().fit(X, y)
13only = LinearRegression().fit(df[["size"]], y)
14print(both.coef_.round(3), round(both.intercept_, 2)) # [ 1.998 -2.811] 45.73
15print(only.coef_.round(3), round(only.intercept_, 2)) # [0.8] 92.75
16print(round(both.score(X, y), 3), round(only.score(df[["size"]], y), 3)) # 0.952 0.741
17print(both.predict(pd.DataFrame({"size": [85], "age": [15]})).round(2)) # [173.4]
18
19# Standard errors and 95% intervals (what statsmodels' OLS summary would print):
20n, p = X.shape
21A = np.c_[np.ones(n), X]
22res = y - both.predict(X)
23s2 = res @ res / (n - p - 1)
24se = np.sqrt(np.diag(s2 * np.linalg.inv(A.T @ A)))[1:]
25t = stats.t.ppf(0.975, n - p - 1)
26for name, c, s in zip(X.columns, both.coef_, se):
27 print(f"{name}: {c:.3f} ± {t * s:.3f}") # size: 1.998 ± 0.696, age: -2.811 ± 1.539
scikit-learn on the same data: joint vs size-only fit, prediction, and 95% intervals.

Use when

  • +You need a fast, transparent baseline whose coefficients can be read as partial effects.
  • +You want to adjust one effect for others (confounders) and attach standard errors to it.
  • +The relationship is roughly additive and linear after sensible transforms, and n is comfortably larger than p.
  • +Prediction must be cheap and auditable: one dot product, every term visible.

Avoid when

  • −Effects are strongly non-linear or interact in ways you cannot enumerate — use trees, boosting or kernels.
  • −p is close to or larger than n, or features are heavily collinear — use ridge, lasso or elastic net.
  • −Outliers or heavy-tailed noise dominate — squared error lets a few points pull the plane (use Huber or quantile regression).
  • −You need causal effects but cannot measure the confounders.

Where it's used

Hedonic pricing

House, car and wine prices decomposed into the value of each attribute — size, age, location dummies — holding the rest fixed.

Econometrics & epidemiology

Estimate one exposure's effect (education on wages, smoking on blood pressure) adjusted for age, sex and other measured confounders.

Marketing mix models

Weekly sales regressed on spend per channel, price and seasonality dummies to split credit across channels.

Engineering calibration

Sensor output modelled from temperature, humidity and supply voltage, then inverted to correct readings.

Check yourself

A full model gives b₁ = 3 and b₂ = −2. Regressing x₂ on x₁ gives slope δ = 0.5. What slope do you get regressing y on x₁ alone (same data)?

Check yourself

Two features have sample correlation 0.9 (and there are no others). By what factor is the variance of each coefficient inflated, compared with uncorrelated features?

Check yourself

You add a column of pure random noise to a regression with n = 100. What happens to the training R²?

Check yourself

In the house model ŷ = 45.73 + 1.998·size − 2.811·age, what does −2.811 mean?

Remember

  • ✓Stack features into XX (with a ones column); least squares gives the normal equations X⊤Xb=X⊤yX^\top Xb = X^\top y and residuals orthogonal to every feature.
  • ✓Each bjb_j is a partial slope: by Frisch–Waugh–Lovell it equals the simple slope of yy on xjx_j after both are residualized on the other features.
  • ✓Leaving out a correlated feature biases a coefficient by exactly bomitted⋅δb_{\text{omitted}}\cdot\delta — enough to flip its sign (age: +1.35 alone, −2.81 jointly).
  • ✓Collinearity does not bias coefficients but inflates their variance by the VIF 1/(1−Rj2)1/(1-R_j^2); predictions inside the data stay fine.
  • ✓Training R² never falls as features are added; compare models with adjusted R² or held-out error, and compare features with standardized coefficients.
← back to ML Algorithms roadmap