# Multiple linear regression

> With several features, ŷ = Xw fits a hyperplane; each coefficient is the effect of its feature with the others held fixed.

Source: https://www.mlfoundry.in/learn/multiple-linear-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 16 min read · updated 2026-10-07

**Multiple linear regression:** predicts a real-valued target from $p$ features at once, $\hat y = b_0 + b_1x_1 + \dots + b_px_p$, fitting all coefficients **jointly** by least squares. Each $b_j$ is the change in $\hat y$ per unit of $x_j$ **with every other feature held fixed** — not the slope you would get by regressing $y$ on $x_j$ alone.

- **Task:** Regression — predict a real number from several features
- **Input → output:** $x\in\mathbb{R}^p$ → $\hat y = b_0 + x^\top b$
- **Learns:** $p+1$ numbers: an intercept and one partial slope per feature
- **Training:** form $X^\top X$ in $O(np^2)$, solve in $O(p^3)$ (QR or Cholesky); or gradient descent at $O(np)$ per step
- **Prediction:** $O(p)$ — one dot product
- **Hyperparameters:** none for plain OLS (just whether to fit an intercept); which features and interactions to include is the real modelling choice
- **Assumes:** $y$ is linear in the (possibly transformed) features; errors independent, mean zero, constant variance; no feature is an exact linear combination of others

## The problem

The **linear regression** article fits a line to one feature. Real targets depend on many things at once — a house price on size, age, location, condition — and those things are **correlated with each other**. Regressing the target on each feature separately then gives answers that are not just imprecise but *wrong in sign*.

In the eight houses of the worked example below, bigger houses also tend to be older (correlation 0.94). Regress price on size alone and each extra m² seems worth only **0.80 $k**. Regress price on age alone and each extra year seems to *add* **1.35 $k**. Fit both together and the picture changes: an extra m² is worth **2.00 $k** at a fixed age, and each year *costs* **2.81 $k** at a fixed size. The one-at-a-time slopes were each soaking up the other feature's effect.

## The idea

Fit one model with all features in it. With two features the model is a **plane** over the $(x_1,x_2)$ floor; with $p$ features, a hyperplane in $p+1$ dimensions. Least squares tilts it in every direction at once until the sum of squared vertical errors is smallest.

The tilt along $x_1$ is $b_1$. Moving along the $x_1$ direction of a plane means $x_2$ does not change — so $b_1$ answers *"two houses of the same age, one 1 m² larger: how much more does it cost?"*. Equivalently (a theorem we derive below), $b_1$ is the slope you get after **removing from both size and price whatever age can explain**, then regressing what is left over. That is what "held fixed" means mechanically.

**Vocabulary**

- **Design matrix X:** $n\times(p+1)$ table: one row per example, a column of 1s for the intercept, then one column per feature
- **Coefficient / partial slope:** $b_j$: change in $\hat y$ per unit of $x_j$ with all other features unchanged
- **Intercept:** $b_0$: prediction when every feature is 0 (often outside the data — e.g. a 0 m² house)
- **Fitted value, residual:** $\hat y_i = x_i^\top b$ and $e_i = y_i-\hat y_i$
- **Hat matrix:** $H = X(X^\top X)^{-1}X^\top$, the projection with $\hat y = Hy$
- **Confounder / omitted variable:** a feature that affects y and is correlated with an included feature; leaving it out biases that feature's coefficient
- **Multicollinearity:** features that are strongly linearly related to each other; the fit is fine but individual coefficients become unstable
- **VIF:** variance inflation factor $1/(1-R_j^2)$: how much correlation with the other features inflates $\operatorname{Var}(b_j)$
- **Dummy (indicator):** a 0/1 column encoding one level of a categorical feature
- **Interaction term:** a product column $x_jx_k$ letting the slope of $x_j$ depend on $x_k$

## The math

Stack the $n$ examples into a design matrix and write the whole model at once:

$$
y = X\beta + \varepsilon,\qquad X=\begin{bmatrix}1 & x_{11} & \cdots & x_{1p}\\ \vdots & \vdots & & \vdots\\ 1 & x_{n1} & \cdots & x_{np}\end{bmatrix}
$$

- $y$: $n$-vector of targets
- $X$: $n\times(p+1)$ design matrix; row $i$ is $x_i^\top$ with a leading 1
- $\beta$: true coefficients $(\beta_0,\dots,\beta_p)$; our estimate is $b$
- $\varepsilon$: noise, mean 0, variance $\sigma^2$, independent across rows

$$
\text{SSE}(b)=\lVert y-Xb\rVert^2 = y^\top y - 2b^\top X^\top y + b^\top X^\top X\, b
$$

- $\text{SSE}$: sum of squared errors (n × the MSE); same minimiser

It is a quadratic bowl in $p+1$ dimensions. Its gradient is $\nabla_b \text{SSE} = -2X^\top(y-Xb)$; setting it to zero gives the **normal equations** and, when $X^\top X$ is invertible, the closed form:

$$
X^\top X\,b = X^\top y \quad\Longrightarrow\quad b=(X^\top X)^{-1}X^\top y
$$

- $X^\top X$: $(p+1)\times(p+1)$ Gram matrix: sums of products of feature pairs
- $X^\top y$: sums of products of each feature with the target

The normal equations say $X^\top e = 0$: the residual vector is **orthogonal to every column of $X$**. So the residuals sum to zero (the ones column) and are uncorrelated with every feature in the sample. Geometrically, $\hat y=Hy$ is the orthogonal projection of $y$ onto the $(p+1)$-dimensional subspace spanned by the columns. $X^\top X$ is invertible exactly when no column is a linear combination of the others; the Hessian $2X^\top X$ is then positive definite and the minimum is unique.

## Two features by hand: center first

Subtracting the means removes the intercept from the system (it is recovered at the end), leaving a $2\times2$ problem in centered sums $S_{jk}=\sum_i (x_{ij}-\bar x_j)(x_{ik}-\bar x_k)$ and $S_{jy}=\sum_i (x_{ij}-\bar x_j)(y_i-\bar y)$:

$$
\begin{bmatrix}S_{11}&S_{12}\\S_{12}&S_{22}\end{bmatrix}\begin{bmatrix}b_1\\b_2\end{bmatrix}=\begin{bmatrix}S_{1y}\\S_{2y}\end{bmatrix}\;\Rightarrow\; b_1=\frac{S_{22}S_{1y}-S_{12}S_{2y}}{D},\;\; b_2=\frac{S_{11}S_{2y}-S_{12}S_{1y}}{D}
$$

- $D$: determinant $S_{11}S_{22}-S_{12}^2$; near 0 when the features are nearly collinear
- $b_0$: recovered as $\bar y - b_1\bar x_1 - b_2\bar x_2$ (the plane passes through the means)

If the features are uncorrelated ($S_{12}=0$) this collapses to $b_1=S_{1y}/S_{11}$ — the simple-regression slope. Only correlation between features makes the joint fit differ from the one-at-a-time fits.

## What "held fixed" means: Frisch–Waugh–Lovell

$$
b_1=\frac{\sum_i \tilde x_{i1}\,\tilde y_i}{\sum_i \tilde x_{i1}^2},\qquad \tilde x_1 = x_1 - \widehat{x_1\mid x_2},\quad \tilde y = y-\widehat{y\mid x_2}
$$

- $\tilde x_1$: residual of regressing $x_1$ on the other features: the part of size that age cannot predict
- $\tilde y$: residual of regressing $y$ on the other features

**Derivation.** Write the fit as $y = b_1x_1 + Z c + e$ where $Z$ holds the intercept and every other feature, and let $M = I - Z(Z^\top Z)^{-1}Z^\top$ be the "residual-maker" that removes anything $Z$ explains ($MZ=0$, $M=M^\top=M^2$). Multiply by $M$: $My = b_1 Mx_1 + Me$. Because $e$ is orthogonal to every column of $X$, it is orthogonal to $Z$, so $Me=e$; then $(Mx_1)^\top e = x_1^\top Me = x_1^\top e = 0$. Taking the dot product of $My = b_1 Mx_1 + e$ with $Mx_1$ leaves $(Mx_1)^\top My = b_1\,(Mx_1)^\top Mx_1$, which is the formula with $\tilde x_1 = Mx_1$, $\tilde y = My$. A multiple-regression coefficient is a simple-regression slope **on residualized data**.

## Omitted-variable bias

$$
b^{\text{simple}}_1 = b_1 + b_2\,\delta,\qquad \delta=\frac{S_{12}}{S_{11}}
$$

- $b^{\text{simple}}_1$: slope from regressing $y$ on $x_1$ alone
- $\delta$: slope from regressing the omitted $x_2$ on $x_1$

Substitute $S_{1y} = b_1S_{11}+b_2S_{12}$ (the first normal equation) into $b^{\text{simple}}_1 = S_{1y}/S_{11}$. This is an exact identity on any sample. The bias $b_2\delta$ vanishes only if the omitted feature has no effect ($b_2=0$) **or** is uncorrelated with the included one ($\delta=0$). In the houses: $\delta=0.426$ years per m², so $0.80 = 2.00 + (-2.81)(0.426)$.

## How sure are we about each coefficient?

$$
\operatorname{Var}(b)=\sigma^2(X^\top X)^{-1},\qquad \operatorname{Var}(b_j)=\frac{\sigma^2}{S_{jj}\,(1-R_j^2)},\qquad \hat\sigma^2=\frac{\text{SSE}}{n-p-1}
$$

- $R_j^2$: $R^2$ from regressing $x_j$ on the other features
- $\tfrac{1}{1-R_j^2}$: the VIF: the variance multiplier caused by collinearity
- $n-p-1$: residual degrees of freedom ($p+1$ parameters were fitted)

The second form follows from FWL: $b_j$ is a simple slope on $\tilde x_j$, whose variance is $\sigma^2/\sum\tilde x_{ij}^2$, and $\sum\tilde x_{ij}^2 = S_{jj}(1-R_j^2)$. So collinearity does not bias anything — it shrinks the leftover variation $\tilde x_j$ the coefficient can learn from, and the standard error grows by $\sqrt{\text{VIF}}$. Here the two features correlate at $r=0.94$, so $\text{VIF}=1/(1-0.887)=8.86$ and each standard error is about 3× what it would be with uncorrelated features.

$$
R^2 = 1-\frac{\text{SSE}}{\text{SST}},\qquad R^2_{\text{adj}} = 1-(1-R^2)\frac{n-1}{n-p-1}
$$

- $\text{SST}$: $\sum(y_i-\bar y)^2$, the error of predicting the mean
- $R^2_{\text{adj}}$: R² penalized for the number of features; it can fall when a useless feature is added

Adding a column can never increase the minimum SSE (the old solution, with the new coefficient at 0, is still available), so plain $R^2$ never falls as features are added — even pure noise features. Adjusted $R^2$, held-out error, or cross-validation are the honest comparisons.

> **Hyperparameters:** Plain OLS has none: the closed form is the unique minimiser. The "knobs" are modelling decisions — which features, which transforms, which interactions — and, once $p$ is large or features are collinear, a penalty: $\lambda\lVert b\rVert_2^2$ gives **ridge**, $b=(X^\top X+\lambda I)^{-1}X^\top y$, which keeps the system invertible; $\lambda\lVert b\rVert_1$ gives **lasso**. Curved relationships are handled by adding powers as extra columns — see **polynomial regression**.

## Worked example: price from size and age

Eight houses. Size and age rise together (correlation 0.94), which is exactly the situation where one-feature-at-a-time thinking fails.

Eight houses. Means: size 85 m², age 19.5 years, price 160.75 $k.

| House | Size x₁ (m²) | Age x₂ (years) | Price y ($k) |
| --- | --- | --- | --- |
| 1 | 50 | 5 | 138 |
| 2 | 60 | 12 | 133 |
| 3 | 70 | 8 | 161 |
| 4 | 80 | 20 | 139 |
| 5 | 90 | 18 | 173 |
| 6 | 100 | 30 | 163 |
| 7 | 110 | 26 | 195 |
| 8 | 120 | 37 | 184 |

**Fitting the plane by hand**

1. **Means.** $\bar x_1 = 85$, $\bar x_2 = 19.5$, $\bar y = 160.75$.
2. **Centered sums.** $S_{11}=4200$, $S_{22}=860$, $S_{12}=1790$, $S_{1y}=3360$, $S_{2y}=1159$ (and $\text{SST}=S_{yy}=3629.5$).
3. **Determinant.** $D = 4200\cdot860 - 1790^2 = 3{,}612{,}000-3{,}204{,}100 = 407{,}900$.
4. **Size coefficient.** $b_1 = \dfrac{860\cdot3360 - 1790\cdot1159}{407{,}900} = \dfrac{2{,}889{,}600-2{,}074{,}610}{407{,}900}=\dfrac{814{,}990}{407{,}900} = 1.998$ $k per m².
5. **Age coefficient.** $b_2 = \dfrac{4200\cdot1159-1790\cdot3360}{407{,}900}=\dfrac{4{,}867{,}800-6{,}014{,}400}{407{,}900} = -2.811$ $k per year.
6. **Intercept.** $b_0 = 160.75 - 1.998(85) - (-2.811)(19.5) = 160.75-169.83+54.81 = 45.73$.
7. **Fit quality.** $\text{SSE}=174.10$, so $R^2 = 1-174.10/3629.5 = 0.952$ and $R^2_{\text{adj}} = 1-0.048\cdot\tfrac{7}{5} = 0.933$; $\hat\sigma^2 = 174.10/5 = 34.82$.
8. **Compare.** Size alone: $b^{\text{simple}}_1 = S_{1y}/S_{11} = 3360/4200 = 0.80$, $R^2 = 0.741$. Check the bias identity: $\delta = 1790/4200 = 0.426$ and $1.998 + (-2.811)(0.426) = 0.80$. ✓
9. **Predict** a new 85 m², 15-year-old house: $\hat y = 45.73 + 1.998(85) - 2.811(15) = 45.73 + 169.83 - 42.16 = 173.40$ $k. The size-only model says 160.75 $k (85 m² is the mean size) — it cannot see that this house is 4.5 years younger than average, worth $4.5\times2.811=12.65$ $k — exactly the gap between the two predictions.

Fitted values and residuals of ŷ = 45.73 + 1.998·size − 2.811·age. The residuals sum to 0 and are orthogonal to both feature columns, as the normal equations require. House 4 (cheap for its size and age) has the largest residual.

| House | ŷ ($k) | Residual y − ŷ | Size residual x̃₁ (FWL) | Price residual ỹ (FWL) |
| --- | --- | --- | --- | --- |
| 1 | 131.58 | 6.42 | −4.82 | −3.21 |
| 2 | 131.88 | 1.12 | −9.39 | −17.64 |
| 3 | 163.11 | −2.11 | 8.94 | 15.75 |
| 4 | 149.35 | −10.35 | −6.04 | −22.42 |
| 5 | 174.96 | −1.96 | 8.12 | 14.27 |
| 6 | 161.20 | 1.80 | −6.85 | −11.90 |
| 7 | 192.43 | 2.57 | 11.47 | 25.49 |
| 8 | 181.49 | 2.51 | −1.42 | −0.33 |

The last two columns are FWL in action: regress size on age (slope $1790/860=2.081$ m² per year) and price on age (slope $1159/860 = 1.348$ $k per year), keep the residuals, and the slope of $\tilde y$ on $\tilde x_1$ is again exactly $1.998$. With uncertainty: $b_1 = 1.998 \pm 0.696$ and $b_2 = -2.811\pm1.539$ (95% intervals, standard errors 0.271 and 0.599, $t$ with 5 degrees of freedom) — wide, because VIF $= 8.86$.

> Figure: The eight houses, coloured by age. (1) Ignoring age, the best line has slope 0.80 (R² 0.741). (2) The joint fit is a plane; slicing it at ages 10, 20 and 30 gives parallel lines of slope 2.00, 28.1 $k apart per 10 years (R² 0.952). (3) Partialling out age moves every house to its FWL residual coordinates, and the line through them has slope exactly 2.00. Drag any house up or down in views 1–2: all fits, the solve and the omitted-variable identity update live.

> **Simpson's paradox for slopes:** Age alone has a *positive* slope (+1.35 $k/year) because old houses here are big houses. Within any fixed size the effect is negative (−2.81). Neither number is "wrong" — they answer different questions. The marginal slope describes the population as it is; the partial slope describes what changes when one feature moves and the others do not. Most business and science questions want the second.

## The algorithm, step by step

**Ordinary least squares with p features**

1. **Build X.** One row per example; prepend a column of 1s; one-hot encode categoricals, dropping one level per feature to avoid exact collinearity with the intercept.
2. **Check rank.** If a column is a linear combination of others (duplicate unit, all dummies kept, a sum column), $X^\top X$ is singular — drop or merge columns.
3. **Solve.** Compute a QR decomposition $X = QR$ and back-substitute $Rb = Q^\top y$ ($O(np^2)$). This avoids forming $X^\top X$, which squares the condition number. (Cholesky on $X^\top X$ is faster but less stable; SVD handles rank deficiency.)
4. **Diagnose.** Residuals $e = y - Xb$; $R^2$, adjusted $R^2$; $\hat\sigma^2 = \lVert e\rVert^2/(n-p-1)$; standard errors from $\hat\sigma^2(X^\top X)^{-1}$; VIFs; residual-vs-fitted plots for curvature and changing spread.
5. **Predict.** $\hat y = b_0 + x^\top b$ — one dot product per new example.

**Complexity.** Training is $O(np^2 + p^3)$ time and $O(p^2)$ extra memory for the Gram matrix; with $n$ in the millions and $p$ in the hundreds this is seconds. When $p$ reaches tens of thousands, or data arrives in a stream, switch to (stochastic) gradient descent on the same loss, $b \leftarrow b + \tfrac{2\eta}{n}X^\top(y - Xb)$, at $O(np)$ per pass — standardize features first, or the bowl is long and thin and descent zig-zags.

## Variants and practical choices

Things you actually decide when fitting a multiple regression.

| Choice | What it does | When |
| --- | --- | --- |
| Dummy variables | a 0/1 column per category level (minus one); its coefficient is the shift vs the dropped level, others fixed | any categorical feature |
| Interaction x₁·x₂ | slope of x₁ becomes b₁ + b₁₂x₂ | the effect of one feature depends on another |
| Log transform of y | coefficients become approximate % changes | multiplicative effects, right-skewed targets (prices, incomes) |
| Standardize features | coefficients in units of 1 SD; comparable magnitudes; better GD conditioning | comparing importance, gradient descent, before ridge/lasso |
| Ridge / lasso / elastic net | penalize coefficient size to stabilise or select | collinearity, p close to n, many weak features |
| Robust (HC) standard errors | valid intervals when error variance changes with x | residual spread fans out |
| Weighted least squares | minimise Σ wᵢ eᵢ² | known unequal noise per row |

Standardized coefficients make the houses comparable: $b_j\cdot s_{x_j}/s_y$ gives $2.15$ for size and $-1.37$ for age — one SD of size moves price by more than one SD of age does, in either direction.

> **Coefficient size is not importance:** A coefficient's magnitude depends on the feature's units: measure age in decades and $b_2$ becomes $-28.1$ with the same fit. Compare standardized coefficients, or better, the drop in held-out performance when a feature is removed.

> **Collinearity: right predictions, unstable coefficients:** With VIF near 9, moving a single house by a few $k visibly swings $b_1$ and $b_2$ in opposite directions (try it in the figure), while predictions inside the data cloud barely move. If you need to *interpret* coefficients, check VIFs (rules of thumb flag 5–10), collect data that breaks the correlation, combine the features, or use ridge. If you only need predictions near the training data, collinearity is mostly harmless.

> **"Held fixed" is not "caused by":** $b_j$ is adjusted only for the features you included. Any omitted confounder biases it by $b_{\text{omitted}}\cdot\delta$. And "holding others fixed" may describe combinations that never occur (a 120 m² house that is brand new, if none exist) — the model extrapolates linearly there.

## In code

```python
import numpy as np

def fit_ols(X, y):
    """Least squares with an intercept. Returns (w, b)."""
    A = np.c_[np.ones(len(X)), X]                # design matrix with a ones column
    beta, *_ = np.linalg.lstsq(A, y, rcond=None)  # QR/SVD: stable even if XᵀX is ill-conditioned
    return beta[1:], beta[0]

def summary(X, y, w, b):
    n, p = X.shape
    A = np.c_[np.ones(n), X]
    r = y - (X @ w + b)
    sse, sst = r @ r, ((y - y.mean()) ** 2).sum()
    r2 = 1 - sse / sst
    adj = 1 - (1 - r2) * (n - 1) / (n - p - 1)
    s2 = sse / (n - p - 1)                         # unbiased noise variance
    se = np.sqrt(np.diag(s2 * np.linalg.inv(A.T @ A)))
    return r2, adj, se

def vif(X):
    out = []
    for j in range(X.shape[1]):
        others = np.delete(X, j, axis=1)
        w, b = fit_ols(others, X[:, j])
        r = X[:, j] - (others @ w + b)
        r2 = 1 - (r @ r) / ((X[:, j] - X[:, j].mean()) ** 2).sum()
        out.append(1 / (1 - r2))
    return np.array(out)

size  = np.array([50, 60, 70, 80, 90, 100, 110, 120.])
age   = np.array([5, 12, 8, 20, 18, 30, 26, 37.])
price = np.array([138, 133, 161, 139, 173, 163, 195, 184.])
X = np.c_[size, age]

w, b = fit_ols(X, price)
r2, adj, se = summary(X, price, w, b)
print("w =", w.round(3), " b =", round(b, 2))          # [ 1.998 -2.811]  45.73
print("R2 =", round(r2, 3), " adj R2 =", round(adj, 3))  # 0.952  0.933
print("SE (b, size, age) =", se.round(3))              # [12.828  0.271  0.599]
print("VIF =", vif(X).round(2))                        # [8.86 8.86]
print("price(85 m2, 15 y) =", round(np.array([85, 15]) @ w + b, 2))  # 173.4
```

From scratch with NumPy: least squares, R², standard errors and VIF on the eight houses.

```python
import numpy as np, pandas as pd
from scipy import stats
from sklearn.linear_model import LinearRegression

df = pd.DataFrame({
    "size":  [50, 60, 70, 80, 90, 100, 110, 120],
    "age":   [5, 12, 8, 20, 18, 30, 26, 37],
    "price": [138, 133, 161, 139, 173, 163, 195, 184],
})
X, y = df[["size", "age"]], df.price

both = LinearRegression().fit(X, y)
only = LinearRegression().fit(df[["size"]], y)
print(both.coef_.round(3), round(both.intercept_, 2))       # [ 1.998 -2.811] 45.73
print(only.coef_.round(3), round(only.intercept_, 2))       # [0.8] 92.75
print(round(both.score(X, y), 3), round(only.score(df[["size"]], y), 3))  # 0.952 0.741
print(both.predict(pd.DataFrame({"size": [85], "age": [15]})).round(2))  # [173.4]

# Standard errors and 95% intervals (what statsmodels' OLS summary would print):
n, p = X.shape
A = np.c_[np.ones(n), X]
res = y - both.predict(X)
s2 = res @ res / (n - p - 1)
se = np.sqrt(np.diag(s2 * np.linalg.inv(A.T @ A)))[1:]
t = stats.t.ppf(0.975, n - p - 1)
for name, c, s in zip(X.columns, both.coef_, se):
    print(f"{name}: {c:.3f} ± {t * s:.3f}")              # size: 1.998 ± 0.696, age: -2.811 ± 1.539
```

scikit-learn on the same data: joint vs size-only fit, prediction, and 95% intervals.

> **For inference, use statsmodels:** scikit-learn's `LinearRegression` returns only coefficients. `statsmodels.formula.api.ols("price ~ size + age", df).fit().summary()` prints the same estimates with standard errors, t-tests, intervals and R² / adjusted R² in one table, and accepts `C(category)` and `a:b` interactions in the formula.

**Strengths**

- You need a fast, transparent baseline whose coefficients can be read as partial effects.
- You want to adjust one effect for others (confounders) and attach standard errors to it.
- The relationship is roughly additive and linear after sensible transforms, and n is comfortably larger than p.
- Prediction must be cheap and auditable: one dot product, every term visible.

**Limitations**

- Effects are strongly non-linear or interact in ways you cannot enumerate — use trees, boosting or kernels.
- p is close to or larger than n, or features are heavily collinear — use ridge, lasso or elastic net.
- Outliers or heavy-tailed noise dominate — squared error lets a few points pull the plane (use Huber or quantile regression).
- You need causal effects but cannot measure the confounders.

- **Hedonic pricing:** House, car and wine prices decomposed into the value of each attribute — size, age, location dummies — holding the rest fixed.
- **Econometrics & epidemiology:** Estimate one exposure's effect (education on wages, smoking on blood pressure) adjusted for age, sex and other measured confounders.
- **Marketing mix models:** Weekly sales regressed on spend per channel, price and seasonality dummies to split credit across channels.
- **Engineering calibration:** Sensor output modelled from temperature, humidity and supply voltage, then inverted to correct readings.

**Question:** A full model gives b₁ = 3 and b₂ = −2. Regressing x₂ on x₁ gives slope δ = 0.5. What slope do you get regressing y on x₁ alone (same data)?

- A. 3
- B. 2
- C. 4
- D. −1

**Answer:** B. 2 — Omitted-variable identity: $b_1^{\text{simple}} = b_1 + b_2\delta = 3 + (-2)(0.5) = 2$. It is exact on the sample, not an approximation.

**Question:** Two features have sample correlation 0.9 (and there are no others). By what factor is the variance of each coefficient inflated, compared with uncorrelated features?

- A. 1.9
- B. 5.26
- C. 10
- D. 0.19

**Answer:** B. 5.26 — With two features $R_j^2 = r^2 = 0.81$, so $\text{VIF} = 1/(1-0.81) = 5.26$; standard errors grow by $\sqrt{5.26}\approx 2.3$.

**Question:** You add a column of pure random noise to a regression with n = 100. What happens to the training R²?

- A. It can go up or down
- B. It cannot decrease — it stays the same or rises slightly
- C. It always falls, because noise hurts
- D. It is unchanged exactly

**Answer:** B. It cannot decrease — it stays the same or rises slightly — Setting the new coefficient to 0 reproduces the old fit, so the minimum SSE can only stay equal or fall. Adjusted R² (or validation error) is what penalizes the useless column.

**Question:** In the house model ŷ = 45.73 + 1.998·size − 2.811·age, what does −2.811 mean?

- A. Older houses in the data are cheaper on average
- B. Among houses of the same size, each extra year of age predicts 2.81 $k lower price
- C. Age causes a 2.81 $k loss per year
- D. Age explains 2.81% of the variance

**Answer:** B. Among houses of the same size, each extra year of age predicts 2.81 $k lower price — It is a partial slope: size held fixed. Marginally, older houses here are actually pricier (slope +1.35) because they are bigger. And it is a prediction statement, not a causal one, unless every confounder is in the model.

**Key points**

- Stack features into $X$ (with a ones column); least squares gives the normal equations $X^\top Xb = X^\top y$ and residuals orthogonal to every feature.
- Each $b_j$ is a partial slope: by Frisch–Waugh–Lovell it equals the simple slope of $y$ on $x_j$ after both are residualized on the other features.
- Leaving out a correlated feature biases a coefficient by exactly $b_{\text{omitted}}\cdot\delta$ — enough to flip its sign (age: +1.35 alone, −2.81 jointly).
- Collinearity does not bias coefficients but inflates their variance by the VIF $1/(1-R_j^2)$; predictions inside the data stay fine.
- Training R² never falls as features are added; compare models with adjusted R² or held-out error, and compare features with standardized coefficients.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
