Linear regression

Minimize squared error; the closed-form solution is w = (XᵀX)⁻¹Xᵀy.

17 min read1747 wordscore13 viewsupdated 2026-10-07

Definition

Linear regression predicts a real number as a straight-line function of the input, y^=wx+b\hat y = wx + b (or y^=w⊤x+b\hat y = w^\top x + b with several features), choosing the slope and intercept that make the mean squared error on the training data as small as possible. That choice has an exact, closed-form answer.

At a glance

Task
Regression: predict a continuous number (rent, demand, temperature, dose response)
Input → output
feature xx (or vector x∈Rdx\in\mathbb{R}^d) → y^∈R\hat y\in\mathbb{R}
Learns
a slope per feature and one intercept: d+1d+1 numbers in total
Loss
mean squared error 1n∑i(yi−y^i)2\tfrac1n\sum_i (y_i-\hat y_i)^2, a convex bowl with a single minimum
Training
closed form via QR / normal equations in O(nd2+d3)O(nd^2+d^3); or gradient descent at O(nd)O(nd) per step
Prediction
O(d)O(d): one dot product
Hyperparameters
none for plain least squares (just fit_intercept); learning rate and step count if trained by gradient descent
Assumes
the mean of y is linear in x; errors independent, constant variance, roughly Gaussian (needed for intervals and tests, not for the fit itself)

The problem

You have nn examples (xi,yi)(x_i, y_i) (apartment area and monthly rent, say) and want a rule that predicts yy for an xx you have not seen. Any line you draw misses most points by some amount. To choose one line you need two things: a single number that scores how badly a line fits, and a procedure that finds the line with the best score.

The obvious scores do not work. The sum of errors ∑i(yi−y^i)\sum_i (y_i-\hat y_i) lets positive and negative misses cancel: every line through the mean point (xˉ,yˉ)(\bar x,\bar y) scores exactly 0, whatever its slope. The sum of absolute errors fixes the cancellation, but it has corners, often several equally good lines, and no formula for the answer. Legendre (1805) and Gauss (1809) chose the sum of squared errors: it is always positive, punishes a big miss much more than several small ones, is smooth everywhere, and its minimum can be written down exactly. Two hundred years later it is still the first model anyone should fit to a regression problem.

The idea

Picture each training point hanging off the line by a vertical spring. The vertical gap is the residual. Square every gap (literally: draw a square on it) and add up the areas. Least squares is the line that makes the total area of those squares as small as possible. Because the total is a quadratic function of the slope and intercept, the landscape of "total area versus line" is a smooth bowl, and the bottom of a bowl is where both partial derivatives are zero. That gives two linear equations, and solving them gives the line.

Vocabulary

Response / target y
the number being predicted (rent)
Feature / predictor x
the input used to predict it (area). With several, see multiple linear regression
Slope w (coefficient)
change in y^\hat y per one-unit increase in xx; here dollars per m²
Intercept b
the prediction at x=0x=0; often outside the data and not meaningful on its own
Fitted value ŷᵢ
the line's prediction at a training point, y^i=wxi+b\hat y_i = wx_i+b
Residual eᵢ
ei=yi−y^ie_i = y_i-\hat y_i: positive if the point lies above the line
RSS / SSE
residual sum of squares ∑iei2\sum_i e_i^2; MSE is RSS/n/n, RMSE is MSE\sqrt{\text{MSE}} (in the units of y)
TSS
total sum of squares ∑i(yi−yˉ)2\sum_i (y_i-\bar y)^2: the RSS of the flat line y^=yˉ\hat y=\bar y
R² (coefficient of determination)
1−RSS/TSS1-\text{RSS}/\text{TSS}: the fraction of y's variance the line explains
Design matrix X
the n×(d+1)n\times(d+1) matrix of features with a leading column of ones for the intercept
Normal equations
X⊤Xθ=X⊤yX^\top X\theta = X^\top y, the zero-gradient conditions; "normal" because they say the residual is perpendicular (normal) to every column of XX
OLS
ordinary least squares: the method itself (as opposed to weighted, regularized or robust variants)

The math

Start with one feature. The model and the loss are:

y^i=wxi+b,L(w,b)=1n∑i=1n(yi−wxi−b)2\hat y_i = w x_i + b,\qquad L(w,b)=\frac1n\sum_{i=1}^{n}\big(y_i - w x_i - b\big)^2
xi, yix_i,\ y_i
feature and target of example ii
w, bw,\ b
slope and intercept, the two parameters
nn
number of training examples
LL
mean squared error (MSE) of the line on the training data

Derivation. LL is a quadratic in (w,b)(w,b) with positive curvature, so its minimum is where both partial derivatives vanish. Write ei=yi−wxi−be_i = y_i - wx_i - b:

∂L∂b=−2n∑iei=0,∂L∂w=−2n∑iei xi=0\frac{\partial L}{\partial b} = -\frac2n\sum_i e_i = 0,\qquad \frac{\partial L}{\partial w} = -\frac2n\sum_i e_i\,x_i = 0
eie_i
residual yi−y^iy_i-\hat y_i

The first condition says the residuals sum to zero, which rearranges to b=yˉ−wxˉb = \bar y - w\bar x: the fitted line always passes through the mean point (xˉ,yˉ)(\bar x,\bar y). Substitute that bb into the second condition, so ei=(yi−yˉ)−w(xi−xˉ)e_i = (y_i-\bar y) - w(x_i-\bar x), and use ∑i(xi−xˉ)xˉ=0\sum_i (x_i-\bar x)\bar x = 0 to replace xix_i by xi−xˉx_i-\bar x:

∑i(xi−xˉ)[(yi−yˉ)−w(xi−xˉ)]=0  ⟹  w∗=SxySxx=∑i(xi−xˉ)(yi−yˉ)∑i(xi−xˉ)2,b∗=yˉ−w∗xˉ\sum_i (x_i-\bar x)\big[(y_i-\bar y)-w(x_i-\bar x)\big]=0 \;\Longrightarrow\; w^*=\frac{S_{xy}}{S_{xx}}=\frac{\sum_i (x_i-\bar x)(y_i-\bar y)}{\sum_i (x_i-\bar x)^2},\qquad b^*=\bar y-w^*\bar x
xˉ, yˉ\bar x,\ \bar y
sample means of x and y
SxyS_{xy}
sum of cross-products of deviations (n × covariance)
SxxS_{xx}
sum of squared deviations of x (n × variance)
w∗, b∗w^*,\ b^*
the least-squares slope and intercept

So the slope is covariance over variance; equivalently w∗=r sy/sxw^* = r\,s_y/s_x with rr the correlation. If xx does not vary (Sxx=0S_{xx}=0) the slope is undefined: no data, no slope.

Matrix form. Stack a column of ones and the features into the design matrix XX and the parameters into θ=(b,w1,…,wd)\theta=(b,w_1,\dots,w_d). The same derivation in one line:

L(θ)=1n∥y−Xθ∥2,∇L=−2nX⊤(y−Xθ)=0  ⟹  X⊤X θ=X⊤y  ⟹  θ∗=(X⊤X)−1X⊤yL(\theta)=\tfrac1n\lVert y-X\theta\rVert^2,\quad \nabla L = -\tfrac2n X^\top(y-X\theta)=0 \;\Longrightarrow\; X^\top X\,\theta = X^\top y \;\Longrightarrow\; \theta^* = (X^\top X)^{-1}X^\top y
XX
n×(d+1)n\times(d+1) design matrix, first column all ones
θ\theta
intercept and slopes stacked into one vector
X⊤XX^\top X
(d+1)×(d+1)(d+1)\times(d+1) Gram matrix; invertible iff the columns of XX are linearly independent

The Hessian is 2nX⊤X\tfrac2n X^\top X, which is positive semidefinite, so LL is convex and any zero-gradient point is a global minimum; it is unique exactly when XX has full column rank. What the coefficients mean once there are several correlated features is the subject of multiple linear regression; here we stay with one feature and the mechanics.

y^=Xθ∗=X(X⊤X)−1X⊤⏟H y,X⊤(y−y^)=0\hat y = X\theta^* = \underbrace{X(X^\top X)^{-1}X^\top}_{H}\,y,\qquad X^\top(y-\hat y)=0
HH
the "hat" matrix: orthogonal projection onto the column space of XX
y−y^y-\hat y
residual vector, perpendicular to every column of X

Geometry. y^\hat y is the point in the span of the columns of XX closest to yy, i.e. the orthogonal projection. The residual is perpendicular to the ones column (so residuals sum to 0) and to the feature column (so they are uncorrelated with xx). Nothing linear in xx is left in them to explain.

How bad is a wrong line? Write any line as its height at the mean, c=wxˉ+bc = w\bar x + b, plus its slope. Expanding the square and using the two orthogonality conditions above, all cross terms vanish:

MSE(w,b)=MSE∗⏟irreducible+(c−yˉ)2⏟height off+(w−w∗)2 Var⁡(x)⏟slope off\text{MSE}(w,b)=\underbrace{\text{MSE}^*}_{\text{irreducible}}+\underbrace{(c-\bar y)^2}_{\text{height off}}+\underbrace{(w-w^*)^2\,\operatorname{Var}(x)}_{\text{slope off}}
MSE∗\text{MSE}^*
MSE of the least-squares line, the floor no line can beat
cc
the line's height at xˉ\bar x
Var⁡(x)\operatorname{Var}(x)
Sxx/nS_{xx}/n, the spread of the feature

So the loss is an axis-aligned elliptical bowl in (slope, height-at-mean) coordinates. A slope error costs more when xx is spread out, which is also why slopes are estimated more precisely from spread-out data. The figure below draws exactly this bowl.

R2=1−RSSTSS,σ^2=RSSn−2,SE⁡(w∗)=σ^SxxR^2 = 1-\frac{\text{RSS}}{\text{TSS}},\qquad \hat\sigma^2=\frac{\text{RSS}}{n-2},\qquad \operatorname{SE}(w^*)=\frac{\hat\sigma}{\sqrt{S_{xx}}}
R2R^2
share of the variance of y explained by the line (training data)
σ^\hat\sigma
residual standard deviation; n−2n-2 because two parameters were fitted
SE⁡(w∗)\operatorname{SE}(w^*)
standard error of the slope, for confidence intervals and t-tests

Why squares? Maximum likelihood. Assume yi=wxi+b+εiy_i = wx_i+b+\varepsilon_i with independent εi∼N(0,σ2)\varepsilon_i\sim\mathcal N(0,\sigma^2). The negative log-likelihood is n2log⁡(2πσ2)+12σ2∑iei2\tfrac n2\log(2\pi\sigma^2)+\tfrac{1}{2\sigma^2}\sum_i e_i^2, so for any σ\sigma the most likely line is the one that minimizes RSS. Least squares is MLE under Gaussian noise; with Laplace noise MLE would minimize absolute error instead. Putting a prior on ww turns this into Bayesian regression.

Gradient descent and its one hyperparameter. Without solving the equations, step downhill: θ←θ−η 2nX⊤(Xθ−y)\theta \leftarrow \theta - \eta\,\tfrac2n X^\top(X\theta-y). Along each eigenvector of the Hessian 2nX⊤X\tfrac2n X^\top X with eigenvalue λ\lambda, the error is multiplied by (1−ηλ)(1-\eta\lambda) per step. It shrinks only if 0<η<2/λmax⁡0<\eta<2/\lambda_{\max}, and the slowest direction shrinks by (1−ηλmin⁡)(1-\eta\lambda_{\min}), so the number of steps grows with the condition number λmax⁡/λmin⁡\lambda_{\max}/\lambda_{\min}. After standardizing xx (mean 0, standard deviation 1) the Hessian of a one-feature model becomes 2I2I: every direction shrinks by (1−2η)(1-2\eta), η=0.5\eta=0.5 lands on the answer in one step, η=0.25\eta=0.25 halves the error each step, and η≥1\eta\ge1 never converges.

Worked example: rent from floor area

Eight apartments in one neighbourhood. We fit monthly rent (in dollars) against area (m²) and then price a 75 m² apartment.

AptArea x (m²)Rent y ($)
135760
245830
350950
4601010
5701150
6801220
7901390
81051520
Training data: n=8n=8, ∑x=535\sum x = 535, ∑y=8830\sum y = 8830.
Aptx − x̄y − ȳ(x − x̄)(y − ȳ)(x − x̄)²
1−31.875−343.7510957.031016.02
2−21.875−273.755988.28478.52
3−16.875−153.752594.53284.77
4−6.875−93.75644.5347.27
53.12546.25144.539.77
613.125116.251525.78172.27
723.125286.256619.53534.77
838.125416.2515869.531453.52
Σ0044343.753996.88
Deviations from the means xˉ=66.875\bar x = 66.875, yˉ=1103.75\bar y = 1103.75. The last row holds the sums SxyS_{xy} and SxxS_{xx}.

Fitting by hand

  1. 1Means. xˉ=535/8=66.875\bar x = 535/8 = 66.875 m², yˉ=8830/8=1103.75\bar y = 8830/8 = 1103.75 dollars.
  2. 2Sums of deviations (table above): Sxy=44343.75S_{xy} = 44343.75, Sxx=3996.875S_{xx} = 3996.875.
  3. 3Slope. w∗=44343.75/3996.875=11.0946w^* = 44343.75 / 3996.875 = 11.0946: each extra m² adds about 11.09 dollars to the monthly rent.
  4. 4Intercept. b∗=1103.75−11.0946×66.875=361.80b^* = 1103.75 - 11.0946\times66.875 = 361.80. The fitted line is y^=11.0946 x+361.80\hat y = 11.0946\,x + 361.80.
  5. 5Residuals (table below) sum to 0, as the first normal equation demands, and ∑ieixi=0\sum_i e_i x_i = 0 too. RSS=4411.10\text{RSS} = 4411.10, so MSE=4411.10/8=551.39\text{MSE} = 4411.10/8 = 551.39 and RMSE=23.48\text{RMSE} = 23.48: the line is typically about 23 dollars off.
  6. 6Fit quality. TSS=496387.5\text{TSS} = 496387.5, so R2=1−4411.10/496387.5=0.9911R^2 = 1 - 4411.10/496387.5 = 0.9911: area explains 99.1% of the variance in rent on this sample.
  7. 7Uncertainty of the slope. σ^=4411.10/6=27.11\hat\sigma = \sqrt{4411.10/6} = 27.11, SE⁡(w∗)=27.11/3996.875=0.4289\operatorname{SE}(w^*) = 27.11/\sqrt{3996.875} = 0.4289, t=11.0946/0.4289=25.87t = 11.0946/0.4289 = 25.87 (p=2.2×10−7p = 2.2\times10^{-7}). The slope is clearly not zero.
  8. 8Predict. A 75 m² apartment: y^=361.80+11.0946×75=1193.89\hat y = 361.80 + 11.0946\times75 = 1193.89 dollars. The 95% prediction interval for a single new apartment is about 1123 to 1265 dollars.
Area xRent yFitted ŷResidual e = y − ŷe²
35760750.119.8997.82
45830861.06−31.06964.44
50950916.5333.471120.34
6010101027.47−17.47305.36
7011501138.4211.58134.08
8012201249.37−29.37862.40
9013901360.3129.69881.33
10515201526.73−6.7345.32
Fitted values and residuals of ŷ = 11.0946x + 361.80. Residuals sum to 0; their squares sum to RSS = 4411.10.

See it: squares, the bowl, and the descent

rent ($) vs area (m²) · drag the ● handlesloss bowl · drag ●8001000120014001600406080100area (m²)(x̄, ȳ)51015slope w↑ height at x̄
ŷ = 6.67·x + 633.3MSE = 10951.4R² = 0.8235best: 11.09·x + 361.8 (MSE 551.4)
MSE = 551.4 irreducible + 604.3 (height at x̄ off)² + 9795.7 (slope off)²·Var(x)
fig.The 8 apartments from the worked example. Drag the two orange handles, or the orange dot in the loss bowl on the right, to move the line. The pink squares are the squared residuals, and their average is the MSE. The starting line ŷ = 6.67x + 633.3 has MSE 10951.4 = 551.4 (irreducible) + 604.3 (height at x̄ off) + 9795.7 (slope off, times Var x = 499.6). 'Solve exactly' slides it onto ŷ = 11.09x + 361.8 (MSE 551.4, R² = 0.9911), which passes through (x̄, ȳ). Each 'gradient step' (η = 0.25 on standardized x) halves the distance to the bowl's centre: MSE 10951.4 → 3151.4 → 1201.4 → 713.9 → 592.0. Toggle the outlier (100 m², $800) and the best line tilts down to slope 7.12 with R² 0.4357.

Notice that the bowl's contours are axis-aligned ellipses, wider along the slope axis than you might expect: that is the Var⁡(x)\operatorname{Var}(x) term at work. Dragging the line parallel to itself only moves the dot vertically (height error); rotating it about (xˉ,yˉ)(\bar x,\bar y) only moves it horizontally (slope error). The two kinds of error add without interacting.

The algorithm, step by step

Fit(X, y) by ordinary least squares

  1. 1Build the design matrix. Prepend a column of ones to the n×dn\times d feature matrix (skip it only if you truly want the line forced through the origin).
  2. 2Solve the least-squares problem min⁡θ∥y−Xθ∥2\min_\theta \lVert y - X\theta\rVert^2 with a QR decomposition X=QRX = QR: then Rθ=Q⊤yR\theta = Q^\top y is a triangular system. This is what "np.linalg.lstsq" (SVD) and scikit-learn (SciPy "lstsq") do. Avoid forming and inverting X⊤XX^\top X explicitly.
  3. 3Compute residuals e=y−Xθe = y - X\theta, then RSS, MSE, R2R^2 and, if you need inference, σ^2=RSS/(n−d−1)\hat\sigma^2 = \text{RSS}/(n-d-1).
  4. 4Check diagnostics: plot residuals against fitted values and against each feature, look for curves, funnels and isolated large residuals.
  5. 5Predict y^=θ0+∑jθjxj\hat y = \theta_0 + \sum_j \theta_j x_j for new inputs, ideally only inside the range of the training features.
▸ Linear regression · gradient descentstep 1/22
MSE (log) vs iteration
iteration 0w = 0.000b = 0.000MSE = 0.3010
  1. 1w ← 0, b ← 0
  2. 2repeat:
  3. 3 ŷᵢ ← w·xᵢ + b
  4. 4 rᵢ ← ŷᵢ − yᵢ
  5. 5 w ← w − η·(2/n)Σ rᵢxᵢ
  6. 6 b ← b − η·(2/n)Σ rᵢ
  7. 7until MSE stops falling

Start with w = 0, b = 0: a flat line at ŷ = 0. MSE = 0.3010.

fig.The iterative alternative: gradient descent from w = b = 0 on a different, 24-point dataset. Predict, measure the residuals, and step both parameters against the gradient. It reaches the closed-form optimum (dashed, MSE 0.0077) by iteration 164.

Cost

Normal equations
form X⊤XX^\top X in O(nd2)O(nd^2), solve in O(d3)O(d^3); memory O(d2)O(d^2), so it can stream over rows
QR / SVD (lstsq)
O(nd2)O(nd^2) with a larger constant, numerically stable; the default
Gradient descent
O(nd)O(nd) per full-batch step; number of steps grows with the condition number
SGD / mini-batch
O(d)O(d) per example; scales to data that does not fit in memory
Prediction
O(d)O(d) per input; the model is d+1d+1 numbers

Variants and practical choices

Plain least squares has no hyperparameters to tune, so the practical decisions are about which solver, which loss and which features. Closely related models get their own articles: adding several features is multiple linear regression, adding powers of xx is polynomial regression, adding a penalty λ∥w∥2\lambda\lVert w\rVert^2 or λ∥w∥1\lambda\lVert w\rVert_1 is ridge or lasso regression, a full posterior over ww is Bayesian regression, and a non-parametric local average is KNN regression.

VariantMinimizesUse when
OLS (lstsq / QR)∑ei2\sum e_i^2default; small to medium n, any d that fits in memory
Normal equations∑ei2\sum e_i^2very large n, small d: accumulate X⊤XX^\top X and X⊤yX^\top y in one pass
Gradient descent / SGD∑ei2\sum e_i^2 iterativelyhuge or streaming data, or as part of a larger model; standardize first
Weighted least squares∑viei2\sum v_i e_i^2noise variance differs between points (heteroscedasticity) and you know the weights
Huber regressionsquared loss for small e, absolute for largea few outliers you cannot remove; scikit-learn HuberRegressor
Theil–Sen / RANSACmedian of pairwise slopes / consensus of inliersmany gross outliers
Through the origin∑(yi−wxi)2\sum (y_i - wx_i)^2, w=∑xiyi/∑xi2w=\sum x_iy_i/\sum x_i^2physics says y = 0 at x = 0 (fit_intercept=False)
Ways to fit a straight line, and when to reach for each.
AssumptionSymptom when violatedRemedy
Linearity of the meancurved band in residuals vs fittedtransform x or y (log), add polynomial terms, or use a nonlinear model
Independent errorsresiduals correlated in time or by grouptime-series models, clustered standard errors
Constant variancefunnel-shaped residual plotlog-transform y, weighted least squares, robust (HC) standard errors
Roughly Gaussian errorsskewed or heavy-tailed residual histogrammatters mostly for small-n intervals; bootstrap them, or use a robust loss
No dominant outliersone point with a huge residual or leverageinspect it; Huber / RANSAC; report with and without it
The assumptions behind the standard errors and intervals, and what to do when they fail. The fitted line itself is the best linear fit whatever the noise looks like; these matter for trusting predictions and inference.

In code

python
1import numpy as np
2
3class LinearRegressionOLS:
4 """Least squares via the normal equations, solved stably with lstsq (QR/SVD)."""
5 def fit(self, X, y):
6 X = np.asarray(X, float).reshape(len(y), -1)
7 A = np.c_[np.ones(len(X)), X] # column of ones absorbs the bias
8 theta, *_ = np.linalg.lstsq(A, y, rcond=None)
9 self.intercept_, self.coef_ = theta[0], theta[1:]
10 return self
11
12 def predict(self, X):
13 X = np.asarray(X, float).reshape(-1, len(self.coef_))
14 return X @ self.coef_ + self.intercept_
15
16def fit_simple(x, y):
17 """The textbook one-feature formulas: w = Sxy / Sxx, b = ȳ − w·x̄."""
18 dx, dy = x - x.mean(), y - y.mean()
19 w = (dx * dy).sum() / (dx ** 2).sum()
20 return w, y.mean() - w * x.mean()
21
22def fit_gd(x, y, lr=0.25, steps=50):
23 """Gradient descent on standardized x (well-conditioned), then undo the scaling."""
24 mu, sd = x.mean(), x.std()
25 z = (x - mu) / sd
26 c, s = 0.0, 0.0 # ŷ = c + s·z
27 for _ in range(steps):
28 r = c + s * z - y # residuals
29 c -= lr * 2 * r.mean()
30 s -= lr * 2 * (r * z).mean()
31 w = s / sd
32 return w, c - w * mu
33
34area = np.array([35, 45, 50, 60, 70, 80, 90, 105.])
35rent = np.array([760, 830, 950, 1010, 1150, 1220, 1390, 1520.])
36
37w, b = fit_simple(area, rent)
38print(f"closed form : w = {w:.4f}, b = {b:.2f}")
39m = LinearRegressionOLS().fit(area, rent)
40print(f"lstsq : w = {m.coef_[0]:.4f}, b = {m.intercept_:.2f}")
41print(f"grad descent: w = {fit_gd(area, rent)[0]:.4f}, b = {fit_gd(area, rent)[1]:.2f}")
42res = rent - m.predict(area)
43print(f"MSE = {np.mean(res**2):.2f}, R² = {1 - res @ res / ((rent - rent.mean())**2).sum():.4f}")
44print(f"rent for 75 m²: {m.predict([75])[0]:.2f}")
45
46# closed form : w = 11.0946, b = 361.80
47# lstsq : w = 11.0946, b = 361.80
48# grad descent: w = 11.0946, b = 361.80
49# MSE = 551.39, R² = 0.9911
50# rent for 75 m²: 1193.89
From scratch: the one-feature formulas, a general lstsq solver, and gradient descent on standardized x. All three agree.
python
1import numpy as np
2from scipy import stats
3from sklearn.linear_model import LinearRegression
4from sklearn.metrics import mean_squared_error, r2_score
5
6area = np.array([35, 45, 50, 60, 70, 80, 90, 105.]).reshape(-1, 1)
7rent = np.array([760, 830, 950, 1010, 1150, 1220, 1390, 1520.])
8
9model = LinearRegression().fit(area, rent)
10pred = model.predict(area)
11print(f"w = {model.coef_[0]:.4f}, b = {model.intercept_:.2f}")
12print(f"MSE = {mean_squared_error(rent, pred):.2f}, "
13 f"RMSE = {mean_squared_error(rent, pred) ** 0.5:.2f}, R² = {r2_score(rent, pred):.4f}")
14print(f"rent for 75 m²: {model.predict([[75]])[0]:.2f}")
15
16# Inference: standard error of the slope and a 95% prediction interval
17fit = stats.linregress(area.ravel(), rent)
18print(f"SE(w) = {fit.stderr:.4f}, t = {fit.slope / fit.stderr:.2f}, p = {fit.pvalue:.1e}")
19x, n = area.ravel(), len(rent)
20s = np.sqrt(((rent - pred) ** 2).sum() / (n - 2)) # residual std, n − 2 dof
21se_new = s * np.sqrt(1 + 1 / n + (75 - x.mean()) ** 2 / ((x - x.mean()) ** 2).sum())
22t = stats.t.ppf(0.975, n - 2)
23y75 = model.predict([[75]])[0]
24print(f"95% prediction interval at 75 m²: [{y75 - t * se_new:.0f}, {y75 + t * se_new:.0f}]")
25
26# w = 11.0946, b = 361.80
27# MSE = 551.39, RMSE = 23.48, R² = 0.9911
28# rent for 75 m²: 1193.89
29# SE(w) = 0.4289, t = 25.87, p = 2.2e-07
30# 95% prediction interval at 75 m²: [1123, 1265]
With scikit-learn, plus the slope's standard error and a prediction interval from SciPy.

Use when

  • +You need a fast, transparent baseline: the slope is "dollars per m²", and the whole model fits on a sticky note.
  • +The relationship is close to linear, or becomes linear after a transform such as log⁡y\log y.
  • +You need uncertainty and inference: standard errors, confidence and prediction intervals come almost for free.
  • +Data is small: two parameters rarely overfit, and the exact solution needs no tuning.

Avoid when

  • −The response curves, saturates or has thresholds: residuals will show structure. Use polynomial terms, splines, trees or boosting.
  • −There are gross outliers or heavy-tailed noise: squared error lets a few points dominate. Use Huber, Theil–Sen or RANSAC.
  • −You must predict far outside the training range: a straight line extrapolates without limit.
  • −The target is a category or a probability: use logistic regression, not a line clipped to [0, 1].

Where it's used

Pricing and valuation

Rent or house price per m², used-car price per year of age: the slope is the quoted rate.

Calibration in the lab

Instrument readings against known standards (e.g. absorbance vs concentration, Beer–Lambert) and inverting the line to measure unknowns.

Economics and finance

A stock's beta is the least-squares slope of its returns on the market's returns.

Trend and baseline estimation

Linear trend of temperature or sales over time; the first baseline that any fancier model must beat.

Check yourself

For a least-squares fit that includes an intercept, the residuals always…

Check yourself

In the rent example xˉ=66.875\bar x = 66.875 m² and yˉ=1103.75\bar y = 1103.75. What does the fitted line predict for a 66.875 m² apartment?

Check yourself

You re-express area in units of 10 m² (so every x is divided by 10) and refit. The slope becomes…

Check yourself

Gradient descent on one standardized feature, loss MSE, Hessian 2I2I. Which learning rate fails to converge?

Remember

  • ✓Model y^=wx+b\hat y = wx + b, loss = mean squared error; the loss is a convex bowl with one minimum.
  • ✓Zero gradient gives w∗=Sxy/Sxxw^* = S_{xy}/S_{xx}, b∗=yˉ−w∗xˉb^* = \bar y - w^*\bar x; in matrix form X⊤Xθ=X⊤yX^\top X\theta = X^\top y, so θ∗=(X⊤X)−1X⊤y\theta^* = (X^\top X)^{-1}X^\top y. Solve it with QR / lstsq, not an explicit inverse.
  • ✓Residuals sum to zero, are uncorrelated with xx, and the line passes through (xˉ,yˉ)(\bar x,\bar y): the fit is an orthogonal projection.
  • ✓Least squares is maximum likelihood under Gaussian noise; R2R^2, σ^\hat\sigma and SE⁡(w)\operatorname{SE}(w) quantify fit and uncertainty.
  • ✓Gradient descent needs standardized features and a learning rate below 2/λmax; the closed form needs neither.
  • ✓Squared error is outlier-sensitive and lines extrapolate blindly: plot residuals, and use robust losses or nonlinear models when they show structure.
← back to ML Algorithms roadmap