Definition
Multiple linear regression predicts a real-valued target from features at once, , fitting all coefficients jointly by least squares. Each is the change in per unit of with every other feature held fixed — not the slope you would get by regressing on alone.
At a glance
- Task
- Regression — predict a real number from several features
- Input → output
- →
- Learns
- numbers: an intercept and one partial slope per feature
- Training
- form in , solve in (QR or Cholesky); or gradient descent at per step
- Prediction
- — one dot product
- Hyperparameters
- none for plain OLS (just whether to fit an intercept); which features and interactions to include is the real modelling choice
- Assumes
- is linear in the (possibly transformed) features; errors independent, mean zero, constant variance; no feature is an exact linear combination of others
The problem
The linear regression article fits a line to one feature. Real targets depend on many things at once — a house price on size, age, location, condition — and those things are correlated with each other. Regressing the target on each feature separately then gives answers that are not just imprecise but wrong in sign.
In the eight houses of the worked example below, bigger houses also tend to be older (correlation 0.94). Regress price on size alone and each extra m² seems worth only 0.80 $k. Regress price on age alone and each extra year seems to add 1.35 $k. Fit both together and the picture changes: an extra m² is worth 2.00 $k at a fixed age, and each year costs 2.81 $k at a fixed size. The one-at-a-time slopes were each soaking up the other feature's effect.
The idea
Fit one model with all features in it. With two features the model is a plane over the floor; with features, a hyperplane in dimensions. Least squares tilts it in every direction at once until the sum of squared vertical errors is smallest.
The tilt along is . Moving along the direction of a plane means does not change — so answers "two houses of the same age, one 1 m² larger: how much more does it cost?". Equivalently (a theorem we derive below), is the slope you get after removing from both size and price whatever age can explain, then regressing what is left over. That is what "held fixed" means mechanically.
Vocabulary
- Design matrix X
- table: one row per example, a column of 1s for the intercept, then one column per feature
- Coefficient / partial slope
- : change in per unit of with all other features unchanged
- Intercept
- : prediction when every feature is 0 (often outside the data — e.g. a 0 m² house)
- Fitted value, residual
- and
- Hat matrix
- , the projection with
- Confounder / omitted variable
- a feature that affects y and is correlated with an included feature; leaving it out biases that feature's coefficient
- Multicollinearity
- features that are strongly linearly related to each other; the fit is fine but individual coefficients become unstable
- VIF
- variance inflation factor : how much correlation with the other features inflates
- Dummy (indicator)
- a 0/1 column encoding one level of a categorical feature
- Interaction term
- a product column letting the slope of depend on
The math
Stack the examples into a design matrix and write the whole model at once:
- -vector of targets
- design matrix; row is with a leading 1
- true coefficients ; our estimate is
- noise, mean 0, variance , independent across rows
- sum of squared errors (n × the MSE); same minimiser
It is a quadratic bowl in dimensions. Its gradient is ; setting it to zero gives the normal equations and, when is invertible, the closed form:
- Gram matrix: sums of products of feature pairs
- sums of products of each feature with the target
The normal equations say : the residual vector is orthogonal to every column of . So the residuals sum to zero (the ones column) and are uncorrelated with every feature in the sample. Geometrically, is the orthogonal projection of onto the -dimensional subspace spanned by the columns. is invertible exactly when no column is a linear combination of the others; the Hessian is then positive definite and the minimum is unique.
Two features by hand: center first
Subtracting the means removes the intercept from the system (it is recovered at the end), leaving a problem in centered sums and :
- determinant ; near 0 when the features are nearly collinear
- recovered as (the plane passes through the means)
If the features are uncorrelated () this collapses to — the simple-regression slope. Only correlation between features makes the joint fit differ from the one-at-a-time fits.
What "held fixed" means: Frisch–Waugh–Lovell
- residual of regressing on the other features: the part of size that age cannot predict
- residual of regressing on the other features
Derivation. Write the fit as where holds the intercept and every other feature, and let be the "residual-maker" that removes anything explains (, ). Multiply by : . Because is orthogonal to every column of , it is orthogonal to , so ; then . Taking the dot product of with leaves , which is the formula with , . A multiple-regression coefficient is a simple-regression slope on residualized data.
Omitted-variable bias
- slope from regressing on alone
- slope from regressing the omitted on
Substitute (the first normal equation) into . This is an exact identity on any sample. The bias vanishes only if the omitted feature has no effect () or is uncorrelated with the included one (). In the houses: years per m², so .
How sure are we about each coefficient?
- from regressing on the other features
- the VIF: the variance multiplier caused by collinearity
- residual degrees of freedom ( parameters were fitted)
The second form follows from FWL: is a simple slope on , whose variance is , and . So collinearity does not bias anything — it shrinks the leftover variation the coefficient can learn from, and the standard error grows by . Here the two features correlate at , so and each standard error is about 3× what it would be with uncorrelated features.
- , the error of predicting the mean
- R² penalized for the number of features; it can fall when a useless feature is added
Adding a column can never increase the minimum SSE (the old solution, with the new coefficient at 0, is still available), so plain never falls as features are added — even pure noise features. Adjusted , held-out error, or cross-validation are the honest comparisons.
Worked example: price from size and age
Eight houses. Size and age rise together (correlation 0.94), which is exactly the situation where one-feature-at-a-time thinking fails.
| House | Size x₁ (m²) | Age x₂ (years) | Price y ($k) |
|---|---|---|---|
| 1 | 50 | 5 | 138 |
| 2 | 60 | 12 | 133 |
| 3 | 70 | 8 | 161 |
| 4 | 80 | 20 | 139 |
| 5 | 90 | 18 | 173 |
| 6 | 100 | 30 | 163 |
| 7 | 110 | 26 | 195 |
| 8 | 120 | 37 | 184 |
Fitting the plane by hand
- 1Means. , , .
- 2Centered sums. , , , , (and ).
- 3Determinant. .
- 4Size coefficient. $k per m².
- 5Age coefficient. $k per year.
- 6Intercept. .
- 7Fit quality. , so and ; .
- 8Compare. Size alone: , . Check the bias identity: and . ✓
- 9Predict a new 85 m², 15-year-old house: k (85 m² is the mean size) — it cannot see that this house is 4.5 years younger than average, worth $k — exactly the gap between the two predictions.
| House | ŷ ($k) | Residual y − ŷ | Size residual x̃₁ (FWL) | Price residual ỹ (FWL) |
|---|---|---|---|---|
| 1 | 131.58 | 6.42 | −4.82 | −3.21 |
| 2 | 131.88 | 1.12 | −9.39 | −17.64 |
| 3 | 163.11 | −2.11 | 8.94 | 15.75 |
| 4 | 149.35 | −10.35 | −6.04 | −22.42 |
| 5 | 174.96 | −1.96 | 8.12 | 14.27 |
| 6 | 161.20 | 1.80 | −6.85 | −11.90 |
| 7 | 192.43 | 2.57 | 11.47 | 25.49 |
| 8 | 181.49 | 2.51 | −1.42 | −0.33 |
The last two columns are FWL in action: regress size on age (slope m² per year) and price on age (slope \tilde y\tilde x_11.998b_1 = 1.998 \pm 0.696b_2 = -2.811\pm1.539t= 8.86$.
The algorithm, step by step
Ordinary least squares with p features
- 1Build X. One row per example; prepend a column of 1s; one-hot encode categoricals, dropping one level per feature to avoid exact collinearity with the intercept.
- 2Check rank. If a column is a linear combination of others (duplicate unit, all dummies kept, a sum column), is singular — drop or merge columns.
- 3Solve. Compute a QR decomposition and back-substitute (). This avoids forming , which squares the condition number. (Cholesky on is faster but less stable; SVD handles rank deficiency.)
- 4Diagnose. Residuals ; , adjusted ; ; standard errors from ; VIFs; residual-vs-fitted plots for curvature and changing spread.
- 5Predict. — one dot product per new example.
Complexity. Training is time and extra memory for the Gram matrix; with in the millions and in the hundreds this is seconds. When reaches tens of thousands, or data arrives in a stream, switch to (stochastic) gradient descent on the same loss, , at per pass — standardize features first, or the bowl is long and thin and descent zig-zags.
Variants and practical choices
| Choice | What it does | When |
|---|---|---|
| Dummy variables | a 0/1 column per category level (minus one); its coefficient is the shift vs the dropped level, others fixed | any categorical feature |
| Interaction x₁·x₂ | slope of x₁ becomes b₁ + b₁₂x₂ | the effect of one feature depends on another |
| Log transform of y | coefficients become approximate % changes | multiplicative effects, right-skewed targets (prices, incomes) |
| Standardize features | coefficients in units of 1 SD; comparable magnitudes; better GD conditioning | comparing importance, gradient descent, before ridge/lasso |
| Ridge / lasso / elastic net | penalize coefficient size to stabilise or select | collinearity, p close to n, many weak features |
| Robust (HC) standard errors | valid intervals when error variance changes with x | residual spread fans out |
| Weighted least squares | minimise Σ wᵢ eᵢ² | known unequal noise per row |
Standardized coefficients make the houses comparable: gives for size and for age — one SD of size moves price by more than one SD of age does, in either direction.
In code
1import numpy as np23def fit_ols(X, y):4 """Least squares with an intercept. Returns (w, b)."""5 A = np.c_[np.ones(len(X)), X] # design matrix with a ones column6 beta, *_ = np.linalg.lstsq(A, y, rcond=None) # QR/SVD: stable even if XᵀX is ill-conditioned7 return beta[1:], beta[0]89def summary(X, y, w, b):10 n, p = X.shape11 A = np.c_[np.ones(n), X]12 r = y - (X @ w + b)13 sse, sst = r @ r, ((y - y.mean()) ** 2).sum()14 r2 = 1 - sse / sst15 adj = 1 - (1 - r2) * (n - 1) / (n - p - 1)16 s2 = sse / (n - p - 1) # unbiased noise variance17 se = np.sqrt(np.diag(s2 * np.linalg.inv(A.T @ A)))18 return r2, adj, se1920def vif(X):21 out = []22 for j in range(X.shape[1]):23 others = np.delete(X, j, axis=1)24 w, b = fit_ols(others, X[:, j])25 r = X[:, j] - (others @ w + b)26 r2 = 1 - (r @ r) / ((X[:, j] - X[:, j].mean()) ** 2).sum()27 out.append(1 / (1 - r2))28 return np.array(out)2930size = np.array([50, 60, 70, 80, 90, 100, 110, 120.])31age = np.array([5, 12, 8, 20, 18, 30, 26, 37.])32price = np.array([138, 133, 161, 139, 173, 163, 195, 184.])33X = np.c_[size, age]3435w, b = fit_ols(X, price)36r2, adj, se = summary(X, price, w, b)37print("w =", w.round(3), " b =", round(b, 2)) # [ 1.998 -2.811] 45.7338print("R2 =", round(r2, 3), " adj R2 =", round(adj, 3)) # 0.952 0.93339print("SE (b, size, age) =", se.round(3)) # [12.828 0.271 0.599]40print("VIF =", vif(X).round(2)) # [8.86 8.86]41print("price(85 m2, 15 y) =", round(np.array([85, 15]) @ w + b, 2)) # 173.41import numpy as np, pandas as pd2from scipy import stats3from sklearn.linear_model import LinearRegression45df = pd.DataFrame({6 "size": [50, 60, 70, 80, 90, 100, 110, 120],7 "age": [5, 12, 8, 20, 18, 30, 26, 37],8 "price": [138, 133, 161, 139, 173, 163, 195, 184],9})10X, y = df[["size", "age"]], df.price1112both = LinearRegression().fit(X, y)13only = LinearRegression().fit(df[["size"]], y)14print(both.coef_.round(3), round(both.intercept_, 2)) # [ 1.998 -2.811] 45.7315print(only.coef_.round(3), round(only.intercept_, 2)) # [0.8] 92.7516print(round(both.score(X, y), 3), round(only.score(df[["size"]], y), 3)) # 0.952 0.74117print(both.predict(pd.DataFrame({"size": [85], "age": [15]})).round(2)) # [173.4]1819# Standard errors and 95% intervals (what statsmodels' OLS summary would print):20n, p = X.shape21A = np.c_[np.ones(n), X]22res = y - both.predict(X)23s2 = res @ res / (n - p - 1)24se = np.sqrt(np.diag(s2 * np.linalg.inv(A.T @ A)))[1:]25t = stats.t.ppf(0.975, n - p - 1)26for name, c, s in zip(X.columns, both.coef_, se):27 print(f"{name}: {c:.3f} ± {t * s:.3f}") # size: 1.998 ± 0.696, age: -2.811 ± 1.539Use when
- +You need a fast, transparent baseline whose coefficients can be read as partial effects.
- +You want to adjust one effect for others (confounders) and attach standard errors to it.
- +The relationship is roughly additive and linear after sensible transforms, and n is comfortably larger than p.
- +Prediction must be cheap and auditable: one dot product, every term visible.
Avoid when
- −Effects are strongly non-linear or interact in ways you cannot enumerate — use trees, boosting or kernels.
- −p is close to or larger than n, or features are heavily collinear — use ridge, lasso or elastic net.
- −Outliers or heavy-tailed noise dominate — squared error lets a few points pull the plane (use Huber or quantile regression).
- −You need causal effects but cannot measure the confounders.
Where it's used
Hedonic pricing
House, car and wine prices decomposed into the value of each attribute — size, age, location dummies — holding the rest fixed.
Econometrics & epidemiology
Estimate one exposure's effect (education on wages, smoking on blood pressure) adjusted for age, sex and other measured confounders.
Marketing mix models
Weekly sales regressed on spend per channel, price and seasonality dummies to split credit across channels.
Engineering calibration
Sensor output modelled from temperature, humidity and supply voltage, then inverted to correct readings.
Check yourself
A full model gives b₁ = 3 and b₂ = −2. Regressing x₂ on x₁ gives slope δ = 0.5. What slope do you get regressing y on x₁ alone (same data)?
Check yourself
Two features have sample correlation 0.9 (and there are no others). By what factor is the variance of each coefficient inflated, compared with uncorrelated features?
Check yourself
You add a column of pure random noise to a regression with n = 100. What happens to the training R²?
Check yourself
In the house model ŷ = 45.73 + 1.998·size − 2.811·age, what does −2.811 mean?
Remember
- ✓Stack features into (with a ones column); least squares gives the normal equations and residuals orthogonal to every feature.
- ✓Each is a partial slope: by Frisch–Waugh–Lovell it equals the simple slope of on after both are residualized on the other features.
- ✓Leaving out a correlated feature biases a coefficient by exactly — enough to flip its sign (age: +1.35 alone, −2.81 jointly).
- ✓Collinearity does not bias coefficients but inflates their variance by the VIF ; predictions inside the data stay fine.
- ✓Training R² never falls as features are added; compare models with adjusted R² or held-out error, and compare features with standardized coefficients.