# Linear regression

> Minimize squared error; the closed-form solution is w = (XᵀX)⁻¹Xᵀy.

Source: https://www.mlfoundry.in/learn/linear-regression  
Section: ML Algorithms › Supervised · Regression  
Level: core · 17 min read · updated 2026-10-07

**Linear regression:** predicts a real number as a straight-line function of the input, $\hat y = wx + b$ (or $\hat y = w^\top x + b$ with several features), choosing the slope and intercept that make the **mean squared error** on the training data as small as possible. That choice has an exact, closed-form answer.

- **Task:** Regression: predict a continuous number (rent, demand, temperature, dose response)
- **Input → output:** feature $x$ (or vector $x\in\mathbb{R}^d$) → $\hat y\in\mathbb{R}$
- **Learns:** a slope per feature and one intercept: $d+1$ numbers in total
- **Loss:** mean squared error $\tfrac1n\sum_i (y_i-\hat y_i)^2$, a convex bowl with a single minimum
- **Training:** closed form via QR / normal equations in $O(nd^2+d^3)$; or gradient descent at $O(nd)$ per step
- **Prediction:** $O(d)$: one dot product
- **Hyperparameters:** none for plain least squares (just `fit_intercept`); learning rate and step count if trained by gradient descent
- **Assumes:** the mean of y is linear in x; errors independent, constant variance, roughly Gaussian (needed for intervals and tests, not for the fit itself)

## The problem

You have $n$ examples $(x_i, y_i)$ (apartment area and monthly rent, say) and want a rule that predicts $y$ for an $x$ you have not seen. Any line you draw misses most points by some amount. To choose *one* line you need two things: a single number that scores how badly a line fits, and a procedure that finds the line with the best score.

The obvious scores do not work. The **sum of errors** $\sum_i (y_i-\hat y_i)$ lets positive and negative misses cancel: every line through the mean point $(\bar x,\bar y)$ scores exactly 0, whatever its slope. The **sum of absolute errors** fixes the cancellation, but it has corners, often several equally good lines, and no formula for the answer. Legendre (1805) and Gauss (1809) chose the **sum of squared errors**: it is always positive, punishes a big miss much more than several small ones, is smooth everywhere, and its minimum can be written down exactly. Two hundred years later it is still the first model anyone should fit to a regression problem.

## The idea

Picture each training point hanging off the line by a vertical spring. The vertical gap is the **residual**. Square every gap (literally: draw a square on it) and add up the areas. Least squares is the line that makes the total area of those squares as small as possible. Because the total is a quadratic function of the slope and intercept, the landscape of "total area versus line" is a smooth bowl, and the bottom of a bowl is where both partial derivatives are zero. That gives two linear equations, and solving them gives the line.

**Vocabulary**

- **Response / target y:** the number being predicted (rent)
- **Feature / predictor x:** the input used to predict it (area). With several, see *multiple linear regression*
- **Slope w (coefficient):** change in $\hat y$ per one-unit increase in $x$; here dollars per m²
- **Intercept b:** the prediction at $x=0$; often outside the data and not meaningful on its own
- **Fitted value ŷᵢ:** the line's prediction at a training point, $\hat y_i = wx_i+b$
- **Residual eᵢ:** $e_i = y_i-\hat y_i$: positive if the point lies above the line
- **RSS / SSE:** residual sum of squares $\sum_i e_i^2$; MSE is RSS$/n$, RMSE is $\sqrt{\text{MSE}}$ (in the units of y)
- **TSS:** total sum of squares $\sum_i (y_i-\bar y)^2$: the RSS of the flat line $\hat y=\bar y$
- **R² (coefficient of determination):** $1-\text{RSS}/\text{TSS}$: the fraction of y's variance the line explains
- **Design matrix X:** the $n\times(d+1)$ matrix of features with a leading column of ones for the intercept
- **Normal equations:** $X^\top X\theta = X^\top y$, the zero-gradient conditions; "normal" because they say the residual is perpendicular (normal) to every column of $X$
- **OLS:** ordinary least squares: the method itself (as opposed to weighted, regularized or robust variants)

## The math

Start with one feature. The model and the loss are:

$$
\hat y_i = w x_i + b,\qquad L(w,b)=\frac1n\sum_{i=1}^{n}\big(y_i - w x_i - b\big)^2
$$

- $x_i,\ y_i$: feature and target of example $i$
- $w,\ b$: slope and intercept, the two parameters
- $n$: number of training examples
- $L$: mean squared error (MSE) of the line on the training data

**Derivation.** $L$ is a quadratic in $(w,b)$ with positive curvature, so its minimum is where both partial derivatives vanish. Write $e_i = y_i - wx_i - b$:

$$
\frac{\partial L}{\partial b} = -\frac2n\sum_i e_i = 0,\qquad \frac{\partial L}{\partial w} = -\frac2n\sum_i e_i\,x_i = 0
$$

- $e_i$: residual $y_i-\hat y_i$

The first condition says the residuals **sum to zero**, which rearranges to $b = \bar y - w\bar x$: the fitted line always passes through the mean point $(\bar x,\bar y)$. Substitute that $b$ into the second condition, so $e_i = (y_i-\bar y) - w(x_i-\bar x)$, and use $\sum_i (x_i-\bar x)\bar x = 0$ to replace $x_i$ by $x_i-\bar x$:

$$
\sum_i (x_i-\bar x)\big[(y_i-\bar y)-w(x_i-\bar x)\big]=0 \;\Longrightarrow\; w^*=\frac{S_{xy}}{S_{xx}}=\frac{\sum_i (x_i-\bar x)(y_i-\bar y)}{\sum_i (x_i-\bar x)^2},\qquad b^*=\bar y-w^*\bar x
$$

- $\bar x,\ \bar y$: sample means of x and y
- $S_{xy}$: sum of cross-products of deviations (n × covariance)
- $S_{xx}$: sum of squared deviations of x (n × variance)
- $w^*,\ b^*$: the least-squares slope and intercept

So the slope is covariance over variance; equivalently $w^* = r\,s_y/s_x$ with $r$ the correlation. If $x$ does not vary ($S_{xx}=0$) the slope is undefined: no data, no slope.

**Matrix form.** Stack a column of ones and the features into the design matrix $X$ and the parameters into $\theta=(b,w_1,\dots,w_d)$. The same derivation in one line:

$$
L(\theta)=\tfrac1n\lVert y-X\theta\rVert^2,\quad \nabla L = -\tfrac2n X^\top(y-X\theta)=0 \;\Longrightarrow\; X^\top X\,\theta = X^\top y \;\Longrightarrow\; \theta^* = (X^\top X)^{-1}X^\top y
$$

- $X$: $n\times(d+1)$ design matrix, first column all ones
- $\theta$: intercept and slopes stacked into one vector
- $X^\top X$: $(d+1)\times(d+1)$ Gram matrix; invertible iff the columns of $X$ are linearly independent

The Hessian is $\tfrac2n X^\top X$, which is positive semidefinite, so $L$ is convex and any zero-gradient point is a global minimum; it is *unique* exactly when $X$ has full column rank. What the coefficients mean once there are several correlated features is the subject of **multiple linear regression**; here we stay with one feature and the mechanics.

$$
\hat y = X\theta^* = \underbrace{X(X^\top X)^{-1}X^\top}_{H}\,y,\qquad X^\top(y-\hat y)=0
$$

- $H$: the "hat" matrix: orthogonal projection onto the column space of $X$
- $y-\hat y$: residual vector, perpendicular to every column of X

**Geometry.** $\hat y$ is the point in the span of the columns of $X$ closest to $y$, i.e. the orthogonal projection. The residual is perpendicular to the ones column (so residuals sum to 0) and to the feature column (so they are uncorrelated with $x$). Nothing linear in $x$ is left in them to explain.

**How bad is a wrong line?** Write any line as its height at the mean, $c = w\bar x + b$, plus its slope. Expanding the square and using the two orthogonality conditions above, all cross terms vanish:

$$
\text{MSE}(w,b)=\underbrace{\text{MSE}^*}_{\text{irreducible}}+\underbrace{(c-\bar y)^2}_{\text{height off}}+\underbrace{(w-w^*)^2\,\operatorname{Var}(x)}_{\text{slope off}}
$$

- $\text{MSE}^*$: MSE of the least-squares line, the floor no line can beat
- $c$: the line's height at $\bar x$
- $\operatorname{Var}(x)$: $S_{xx}/n$, the spread of the feature

So the loss is an axis-aligned elliptical bowl in (slope, height-at-mean) coordinates. A slope error costs more when $x$ is spread out, which is also why slopes are estimated more precisely from spread-out data. The figure below draws exactly this bowl.

$$
R^2 = 1-\frac{\text{RSS}}{\text{TSS}},\qquad \hat\sigma^2=\frac{\text{RSS}}{n-2},\qquad \operatorname{SE}(w^*)=\frac{\hat\sigma}{\sqrt{S_{xx}}}
$$

- $R^2$: share of the variance of y explained by the line (training data)
- $\hat\sigma$: residual standard deviation; $n-2$ because two parameters were fitted
- $\operatorname{SE}(w^*)$: standard error of the slope, for confidence intervals and t-tests

**Why squares? Maximum likelihood.** Assume $y_i = wx_i+b+\varepsilon_i$ with independent $\varepsilon_i\sim\mathcal N(0,\sigma^2)$. The negative log-likelihood is $\tfrac n2\log(2\pi\sigma^2)+\tfrac{1}{2\sigma^2}\sum_i e_i^2$, so for any $\sigma$ the most likely line is the one that minimizes RSS. Least squares is MLE under Gaussian noise; with Laplace noise MLE would minimize absolute error instead. Putting a prior on $w$ turns this into **Bayesian regression**.

**Gradient descent and its one hyperparameter.** Without solving the equations, step downhill: $\theta \leftarrow \theta - \eta\,\tfrac2n X^\top(X\theta-y)$. Along each eigenvector of the Hessian $\tfrac2n X^\top X$ with eigenvalue $\lambda$, the error is multiplied by $(1-\eta\lambda)$ per step. It shrinks only if $0<\eta<2/\lambda_{\max}$, and the slowest direction shrinks by $(1-\eta\lambda_{\min})$, so the number of steps grows with the condition number $\lambda_{\max}/\lambda_{\min}$. After **standardizing** $x$ (mean 0, standard deviation 1) the Hessian of a one-feature model becomes $2I$: every direction shrinks by $(1-2\eta)$, $\eta=0.5$ lands on the answer in one step, $\eta=0.25$ halves the error each step, and $\eta\ge1$ never converges.

## Worked example: rent from floor area

Eight apartments in one neighbourhood. We fit monthly rent (in dollars) against area (m²) and then price a 75 m² apartment.

Training data: $n=8$, $\sum x = 535$, $\sum y = 8830$.

| Apt | Area x (m²) | Rent y ($) |
| --- | --- | --- |
| 1 | 35 | 760 |
| 2 | 45 | 830 |
| 3 | 50 | 950 |
| 4 | 60 | 1010 |
| 5 | 70 | 1150 |
| 6 | 80 | 1220 |
| 7 | 90 | 1390 |
| 8 | 105 | 1520 |

Deviations from the means $\bar x = 66.875$, $\bar y = 1103.75$. The last row holds the sums $S_{xy}$ and $S_{xx}$.

| Apt | x − x̄ | y − ȳ | (x − x̄)(y − ȳ) | (x − x̄)² |
| --- | --- | --- | --- | --- |
| 1 | −31.875 | −343.75 | 10957.03 | 1016.02 |
| 2 | −21.875 | −273.75 | 5988.28 | 478.52 |
| 3 | −16.875 | −153.75 | 2594.53 | 284.77 |
| 4 | −6.875 | −93.75 | 644.53 | 47.27 |
| 5 | 3.125 | 46.25 | 144.53 | 9.77 |
| 6 | 13.125 | 116.25 | 1525.78 | 172.27 |
| 7 | 23.125 | 286.25 | 6619.53 | 534.77 |
| 8 | 38.125 | 416.25 | 15869.53 | 1453.52 |
| **Σ** | 0 | 0 | **44343.75** | **3996.88** |

**Fitting by hand**

1. **Means.** $\bar x = 535/8 = 66.875$ m², $\bar y = 8830/8 = 1103.75$ dollars.
2. **Sums of deviations** (table above): $S_{xy} = 44343.75$, $S_{xx} = 3996.875$.
3. **Slope.** $w^* = 44343.75 / 3996.875 = 11.0946$: each extra m² adds about 11.09 dollars to the monthly rent.
4. **Intercept.** $b^* = 1103.75 - 11.0946\times66.875 = 361.80$. The fitted line is $\hat y = 11.0946\,x + 361.80$.
5. **Residuals** (table below) sum to 0, as the first normal equation demands, and $\sum_i e_i x_i = 0$ too. $\text{RSS} = 4411.10$, so $\text{MSE} = 4411.10/8 = 551.39$ and $\text{RMSE} = 23.48$: the line is typically about 23 dollars off.
6. **Fit quality.** $\text{TSS} = 496387.5$, so $R^2 = 1 - 4411.10/496387.5 = 0.9911$: area explains 99.1% of the variance in rent on this sample.
7. **Uncertainty of the slope.** $\hat\sigma = \sqrt{4411.10/6} = 27.11$, $\operatorname{SE}(w^*) = 27.11/\sqrt{3996.875} = 0.4289$, $t = 11.0946/0.4289 = 25.87$ ($p = 2.2\times10^{-7}$). The slope is clearly not zero.
8. **Predict.** A 75 m² apartment: $\hat y = 361.80 + 11.0946\times75 = 1193.89$ dollars. The 95% prediction interval for a single new apartment is about 1123 to 1265 dollars.

Fitted values and residuals of ŷ = 11.0946x + 361.80. Residuals sum to 0; their squares sum to RSS = 4411.10.

| Area x | Rent y | Fitted ŷ | Residual e = y − ŷ | e² |
| --- | --- | --- | --- | --- |
| 35 | 760 | 750.11 | 9.89 | 97.82 |
| 45 | 830 | 861.06 | −31.06 | 964.44 |
| 50 | 950 | 916.53 | 33.47 | 1120.34 |
| 60 | 1010 | 1027.47 | −17.47 | 305.36 |
| 70 | 1150 | 1138.42 | 11.58 | 134.08 |
| 80 | 1220 | 1249.37 | −29.37 | 862.40 |
| 90 | 1390 | 1360.31 | 29.69 | 881.33 |
| 105 | 1520 | 1526.73 | −6.73 | 45.32 |

> **Read the intercept with care:** $b^* = 361.80$ is the predicted rent of a 0 m² apartment. No such apartment exists and none is in the data (the smallest is 35 m²), so the intercept is only an anchor for the line, not a price.

## See it: squares, the bowl, and the descent

> Figure: The 8 apartments from the worked example. Drag the two orange handles, or the orange dot in the loss bowl on the right, to move the line. The pink squares are the squared residuals, and their average is the MSE. The starting line ŷ = 6.67x + 633.3 has MSE 10951.4 = 551.4 (irreducible) + 604.3 (height at x̄ off) + 9795.7 (slope off, times Var x = 499.6). 'Solve exactly' slides it onto ŷ = 11.09x + 361.8 (MSE 551.4, R² = 0.9911), which passes through (x̄, ȳ). Each 'gradient step' (η = 0.25 on standardized x) halves the distance to the bowl's centre: MSE 10951.4 → 3151.4 → 1201.4 → 713.9 → 592.0. Toggle the outlier (100 m², $800) and the best line tilts down to slope 7.12 with R² 0.4357.

Notice that the bowl's contours are axis-aligned ellipses, wider along the slope axis than you might expect: that is the $\operatorname{Var}(x)$ term at work. Dragging the line *parallel* to itself only moves the dot vertically (height error); rotating it about $(\bar x,\bar y)$ only moves it horizontally (slope error). The two kinds of error add without interacting.

## The algorithm, step by step

**Fit(X, y) by ordinary least squares**

1. **Build the design matrix.** Prepend a column of ones to the $n\times d$ feature matrix (skip it only if you truly want the line forced through the origin).
2. **Solve the least-squares problem** $\min_\theta \lVert y - X\theta\rVert^2$ with a QR decomposition $X = QR$: then $R\theta = Q^\top y$ is a triangular system. This is what "np.linalg.lstsq" (SVD) and scikit-learn (SciPy "lstsq") do. Avoid forming and inverting $X^\top X$ explicitly.
3. **Compute residuals** $e = y - X\theta$, then RSS, MSE, $R^2$ and, if you need inference, $\hat\sigma^2 = \text{RSS}/(n-d-1)$.
4. **Check diagnostics**: plot residuals against fitted values and against each feature, look for curves, funnels and isolated large residuals.
5. **Predict** $\hat y = \theta_0 + \sum_j \theta_j x_j$ for new inputs, ideally only inside the range of the training features.

> Figure: The iterative alternative: gradient descent from w = b = 0 on a different, 24-point dataset. Predict, measure the residuals, and step both parameters against the gradient. It reaches the closed-form optimum (dashed, MSE 0.0077) by iteration 164.

**Cost**

- **Normal equations:** form $X^\top X$ in $O(nd^2)$, solve in $O(d^3)$; memory $O(d^2)$, so it can stream over rows
- **QR / SVD (lstsq):** $O(nd^2)$ with a larger constant, numerically stable; the default
- **Gradient descent:** $O(nd)$ per full-batch step; number of steps grows with the condition number
- **SGD / mini-batch:** $O(d)$ per example; scales to data that does not fit in memory
- **Prediction:** $O(d)$ per input; the model is $d+1$ numbers

> **Unscaled features make gradient descent crawl:** On the raw rent data (area 35 to 105) the Hessian $\tfrac2n X^\top X$ has condition number about 49,500, the largest stable step is $\eta = 0.000201$, and even at 90% of that step, **100,000 iterations** still leave the MSE at 560.50 with intercept 352.27 instead of 361.80. After standardizing area, $\eta = 0.5$ converges in one step. The closed form does not care, but every iterative solver (and every regularized model) does.

## Variants and practical choices

Plain least squares has no hyperparameters to tune, so the practical decisions are about **which solver**, **which loss** and **which features**. Closely related models get their own articles: adding several features is **multiple linear regression**, adding powers of $x$ is **polynomial regression**, adding a penalty $\lambda\lVert w\rVert^2$ or $\lambda\lVert w\rVert_1$ is **ridge** or **lasso regression**, a full posterior over $w$ is **Bayesian regression**, and a non-parametric local average is **KNN regression**.

Ways to fit a straight line, and when to reach for each.

| Variant | Minimizes | Use when |
| --- | --- | --- |
| **OLS (lstsq / QR)** | $\sum e_i^2$ | default; small to medium n, any d that fits in memory |
| Normal equations | $\sum e_i^2$ | very large n, small d: accumulate $X^\top X$ and $X^\top y$ in one pass |
| Gradient descent / SGD | $\sum e_i^2$ iteratively | huge or streaming data, or as part of a larger model; standardize first |
| Weighted least squares | $\sum v_i e_i^2$ | noise variance differs between points (heteroscedasticity) and you know the weights |
| Huber regression | squared loss for small e, absolute for large | a few outliers you cannot remove; scikit-learn `HuberRegressor` |
| Theil–Sen / RANSAC | median of pairwise slopes / consensus of inliers | many gross outliers |
| Through the origin | $\sum (y_i - wx_i)^2$, $w=\sum x_iy_i/\sum x_i^2$ | physics says y = 0 at x = 0 (fit_intercept=False) |

The assumptions behind the standard errors and intervals, and what to do when they fail. The fitted line itself is the best linear fit whatever the noise looks like; these matter for trusting predictions and inference.

| Assumption | Symptom when violated | Remedy |
| --- | --- | --- |
| Linearity of the mean | curved band in residuals vs fitted | transform x or y (log), add polynomial terms, or use a nonlinear model |
| Independent errors | residuals correlated in time or by group | time-series models, clustered standard errors |
| Constant variance | funnel-shaped residual plot | log-transform y, weighted least squares, robust (HC) standard errors |
| Roughly Gaussian errors | skewed or heavy-tailed residual histogram | matters mostly for small-n intervals; bootstrap them, or use a robust loss |
| No dominant outliers | one point with a huge residual or leverage | inspect it; Huber / RANSAC; report with and without it |

> **One bad row can wreck the line:** Add a single mistyped apartment (100 m² renting for 800 dollars) to the eight in the worked example. Because its residual is *squared*, it drags the slope from 11.09 down to 7.12 and $R^2$ falls from 0.9911 to 0.4357. That one row makes the new line's MSE on the eight good apartments 12040.2 instead of 551.4. Plot the data, check residuals, and consider a robust loss if errors like this are possible.

> **Do not extrapolate far:** The line is fitted on 35–105 m². At 300 m² it predicts 3690 dollars, but nothing in the data says rents stay linear that far out, and the prediction interval widens with $(x-\bar x)^2$. A linear model is a local approximation.

## In code

```python
import numpy as np

class LinearRegressionOLS:
    """Least squares via the normal equations, solved stably with lstsq (QR/SVD)."""
    def fit(self, X, y):
        X = np.asarray(X, float).reshape(len(y), -1)
        A = np.c_[np.ones(len(X)), X]               # column of ones absorbs the bias
        theta, *_ = np.linalg.lstsq(A, y, rcond=None)
        self.intercept_, self.coef_ = theta[0], theta[1:]
        return self

    def predict(self, X):
        X = np.asarray(X, float).reshape(-1, len(self.coef_))
        return X @ self.coef_ + self.intercept_

def fit_simple(x, y):
    """The textbook one-feature formulas: w = Sxy / Sxx, b = ȳ − w·x̄."""
    dx, dy = x - x.mean(), y - y.mean()
    w = (dx * dy).sum() / (dx ** 2).sum()
    return w, y.mean() - w * x.mean()

def fit_gd(x, y, lr=0.25, steps=50):
    """Gradient descent on standardized x (well-conditioned), then undo the scaling."""
    mu, sd = x.mean(), x.std()
    z = (x - mu) / sd
    c, s = 0.0, 0.0                               # ŷ = c + s·z
    for _ in range(steps):
        r = c + s * z - y                         # residuals
        c -= lr * 2 * r.mean()
        s -= lr * 2 * (r * z).mean()
    w = s / sd
    return w, c - w * mu

area = np.array([35, 45, 50, 60, 70, 80, 90, 105.])
rent = np.array([760, 830, 950, 1010, 1150, 1220, 1390, 1520.])

w, b = fit_simple(area, rent)
print(f"closed form : w = {w:.4f}, b = {b:.2f}")
m = LinearRegressionOLS().fit(area, rent)
print(f"lstsq       : w = {m.coef_[0]:.4f}, b = {m.intercept_:.2f}")
print(f"grad descent: w = {fit_gd(area, rent)[0]:.4f}, b = {fit_gd(area, rent)[1]:.2f}")
res = rent - m.predict(area)
print(f"MSE = {np.mean(res**2):.2f}, R² = {1 - res @ res / ((rent - rent.mean())**2).sum():.4f}")
print(f"rent for 75 m²: {m.predict([75])[0]:.2f}")

# closed form : w = 11.0946, b = 361.80
# lstsq       : w = 11.0946, b = 361.80
# grad descent: w = 11.0946, b = 361.80
# MSE = 551.39, R² = 0.9911
# rent for 75 m²: 1193.89
```

From scratch: the one-feature formulas, a general lstsq solver, and gradient descent on standardized x. All three agree.

```python
import numpy as np
from scipy import stats
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

area = np.array([35, 45, 50, 60, 70, 80, 90, 105.]).reshape(-1, 1)
rent = np.array([760, 830, 950, 1010, 1150, 1220, 1390, 1520.])

model = LinearRegression().fit(area, rent)
pred = model.predict(area)
print(f"w = {model.coef_[0]:.4f}, b = {model.intercept_:.2f}")
print(f"MSE = {mean_squared_error(rent, pred):.2f}, "
      f"RMSE = {mean_squared_error(rent, pred) ** 0.5:.2f}, R² = {r2_score(rent, pred):.4f}")
print(f"rent for 75 m²: {model.predict([[75]])[0]:.2f}")

# Inference: standard error of the slope and a 95% prediction interval
fit = stats.linregress(area.ravel(), rent)
print(f"SE(w) = {fit.stderr:.4f}, t = {fit.slope / fit.stderr:.2f}, p = {fit.pvalue:.1e}")
x, n = area.ravel(), len(rent)
s = np.sqrt(((rent - pred) ** 2).sum() / (n - 2))           # residual std, n − 2 dof
se_new = s * np.sqrt(1 + 1 / n + (75 - x.mean()) ** 2 / ((x - x.mean()) ** 2).sum())
t = stats.t.ppf(0.975, n - 2)
y75 = model.predict([[75]])[0]
print(f"95% prediction interval at 75 m²: [{y75 - t * se_new:.0f}, {y75 + t * se_new:.0f}]")

# w = 11.0946, b = 361.80
# MSE = 551.39, RMSE = 23.48, R² = 0.9911
# rent for 75 m²: 1193.89
# SE(w) = 0.4289, t = 25.87, p = 2.2e-07
# 95% prediction interval at 75 m²: [1123, 1265]
```

With scikit-learn, plus the slope's standard error and a prediction interval from SciPy.

> **Training R² is not test performance:** $R^2 = 0.9911$ describes how well the line fits the eight apartments it was fitted on. With one feature and eight points the gap to new data is small, but as features are added training $R^2$ can only go up. Judge models on held-out RMSE (or cross-validated $R^2$).

**Strengths**

- You need a fast, transparent baseline: the slope is "dollars per m²", and the whole model fits on a sticky note.
- The relationship is close to linear, or becomes linear after a transform such as $\log y$.
- You need uncertainty and inference: standard errors, confidence and prediction intervals come almost for free.
- Data is small: two parameters rarely overfit, and the exact solution needs no tuning.

**Limitations**

- The response curves, saturates or has thresholds: residuals will show structure. Use polynomial terms, splines, trees or boosting.
- There are gross outliers or heavy-tailed noise: squared error lets a few points dominate. Use Huber, Theil–Sen or RANSAC.
- You must predict far outside the training range: a straight line extrapolates without limit.
- The target is a category or a probability: use logistic regression, not a line clipped to [0, 1].

- **Pricing and valuation:** Rent or house price per m², used-car price per year of age: the slope is the quoted rate.
- **Calibration in the lab:** Instrument readings against known standards (e.g. absorbance vs concentration, Beer–Lambert) and inverting the line to measure unknowns.
- **Economics and finance:** A stock's beta is the least-squares slope of its returns on the market's returns.
- **Trend and baseline estimation:** Linear trend of temperature or sales over time; the first baseline that any fancier model must beat.

> **The mean point and the zero-sum residuals are not coincidences:** Both come from $\partial L/\partial b = 0$. Any model with a free intercept fitted by least squares, including multiple and polynomial regression and the output layer of a regression network, has residuals that average to exactly zero on the training set.

**Question:** For a least-squares fit that includes an intercept, the residuals always…

- A. are all positive
- B. sum to zero
- C. are all equal
- D. sum to n

**Answer:** B. sum to zero — $\partial L/\partial b = -\tfrac2n\sum_i e_i$ must be zero at the optimum. In the worked example the residuals 9.89, −31.06, 33.47, −17.47, 11.58, −29.37, 29.69, −6.73 add to 0.

**Question:** In the rent example $\bar x = 66.875$ m² and $\bar y = 1103.75$. What does the fitted line predict for a 66.875 m² apartment?

- A. 1103.75 dollars
- B. 1193.89 dollars
- C. 361.80 dollars
- D. It depends on the slope

**Answer:** A. 1103.75 dollars — $b^* = \bar y - w^*\bar x$ means $w^*\bar x + b^* = \bar y$: every least-squares line with an intercept passes through $(\bar x,\bar y)$, whatever its slope.

**Question:** You re-express area in units of 10 m² (so every x is divided by 10) and refit. The slope becomes…

- A. 11.0946
- B. 1.10946
- C. 110.946
- D. unchanged R² but a different intercept

**Answer:** C. 110.946 — $w x = (10w)(x/10)$: predictions, residuals, $R^2$ and the intercept are unchanged, so the slope is multiplied by 10, to 110.95 dollars per 10 m². Slopes carry units; compare them only after standardizing.

**Question:** Gradient descent on one standardized feature, loss MSE, Hessian $2I$. Which learning rate fails to converge?

- A. $\eta = 0.1$
- B. $\eta = 0.25$
- C. $\eta = 0.5$
- D. $\eta = 1.0$

**Answer:** D. $\eta = 1.0$ — Each step multiplies the error by $1-2\eta$. At $\eta=0.5$ that factor is 0 (one-step convergence), at 0.25 it is 0.5, at 0.1 it is 0.8, and at $\eta=1$ it is $-1$: the parameters jump back and forth across the minimum forever.

**Key points**

- Model $\hat y = wx + b$, loss = mean squared error; the loss is a convex bowl with one minimum.
- Zero gradient gives $w^* = S_{xy}/S_{xx}$, $b^* = \bar y - w^*\bar x$; in matrix form $X^\top X\theta = X^\top y$, so $\theta^* = (X^\top X)^{-1}X^\top y$. Solve it with QR / lstsq, not an explicit inverse.
- Residuals sum to zero, are uncorrelated with $x$, and the line passes through $(\bar x,\bar y)$: the fit is an orthogonal projection.
- Least squares is maximum likelihood under Gaussian noise; $R^2$, $\hat\sigma$ and $\operatorname{SE}(w)$ quantify fit and uncertainty.
- Gradient descent needs standardized features and a learning rate below 2/λmax; the closed form needs neither.
- Squared error is outlier-sensitive and lines extrapolate blindly: plot residuals, and use robust losses or nonlinear models when they show structure.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
