Definition
Linear regression predicts a real number as a straight-line function of the input, (or with several features), choosing the slope and intercept that make the mean squared error on the training data as small as possible. That choice has an exact, closed-form answer.
At a glance
- Task
- Regression: predict a continuous number (rent, demand, temperature, dose response)
- Input → output
- feature (or vector ) →
- Learns
- a slope per feature and one intercept: numbers in total
- Loss
- mean squared error , a convex bowl with a single minimum
- Training
- closed form via QR / normal equations in ; or gradient descent at per step
- Prediction
- : one dot product
- Hyperparameters
- none for plain least squares (just
fit_intercept); learning rate and step count if trained by gradient descent - Assumes
- the mean of y is linear in x; errors independent, constant variance, roughly Gaussian (needed for intervals and tests, not for the fit itself)
The problem
You have examples (apartment area and monthly rent, say) and want a rule that predicts for an you have not seen. Any line you draw misses most points by some amount. To choose one line you need two things: a single number that scores how badly a line fits, and a procedure that finds the line with the best score.
The obvious scores do not work. The sum of errors lets positive and negative misses cancel: every line through the mean point scores exactly 0, whatever its slope. The sum of absolute errors fixes the cancellation, but it has corners, often several equally good lines, and no formula for the answer. Legendre (1805) and Gauss (1809) chose the sum of squared errors: it is always positive, punishes a big miss much more than several small ones, is smooth everywhere, and its minimum can be written down exactly. Two hundred years later it is still the first model anyone should fit to a regression problem.
The idea
Picture each training point hanging off the line by a vertical spring. The vertical gap is the residual. Square every gap (literally: draw a square on it) and add up the areas. Least squares is the line that makes the total area of those squares as small as possible. Because the total is a quadratic function of the slope and intercept, the landscape of "total area versus line" is a smooth bowl, and the bottom of a bowl is where both partial derivatives are zero. That gives two linear equations, and solving them gives the line.
Vocabulary
- Response / target y
- the number being predicted (rent)
- Feature / predictor x
- the input used to predict it (area). With several, see multiple linear regression
- Slope w (coefficient)
- change in per one-unit increase in ; here dollars per m²
- Intercept b
- the prediction at ; often outside the data and not meaningful on its own
- Fitted value ŷᵢ
- the line's prediction at a training point,
- Residual eᵢ
- : positive if the point lies above the line
- RSS / SSE
- residual sum of squares ; MSE is RSS, RMSE is (in the units of y)
- TSS
- total sum of squares : the RSS of the flat line
- R² (coefficient of determination)
- : the fraction of y's variance the line explains
- Design matrix X
- the matrix of features with a leading column of ones for the intercept
- Normal equations
- , the zero-gradient conditions; "normal" because they say the residual is perpendicular (normal) to every column of
- OLS
- ordinary least squares: the method itself (as opposed to weighted, regularized or robust variants)
The math
Start with one feature. The model and the loss are:
- feature and target of example
- slope and intercept, the two parameters
- number of training examples
- mean squared error (MSE) of the line on the training data
Derivation. is a quadratic in with positive curvature, so its minimum is where both partial derivatives vanish. Write :
- residual
The first condition says the residuals sum to zero, which rearranges to : the fitted line always passes through the mean point . Substitute that into the second condition, so , and use to replace by :
- sample means of x and y
- sum of cross-products of deviations (n × covariance)
- sum of squared deviations of x (n × variance)
- the least-squares slope and intercept
So the slope is covariance over variance; equivalently with the correlation. If does not vary () the slope is undefined: no data, no slope.
Matrix form. Stack a column of ones and the features into the design matrix and the parameters into . The same derivation in one line:
- design matrix, first column all ones
- intercept and slopes stacked into one vector
- Gram matrix; invertible iff the columns of are linearly independent
The Hessian is , which is positive semidefinite, so is convex and any zero-gradient point is a global minimum; it is unique exactly when has full column rank. What the coefficients mean once there are several correlated features is the subject of multiple linear regression; here we stay with one feature and the mechanics.
- the "hat" matrix: orthogonal projection onto the column space of
- residual vector, perpendicular to every column of X
Geometry. is the point in the span of the columns of closest to , i.e. the orthogonal projection. The residual is perpendicular to the ones column (so residuals sum to 0) and to the feature column (so they are uncorrelated with ). Nothing linear in is left in them to explain.
How bad is a wrong line? Write any line as its height at the mean, , plus its slope. Expanding the square and using the two orthogonality conditions above, all cross terms vanish:
- MSE of the least-squares line, the floor no line can beat
- the line's height at
- , the spread of the feature
So the loss is an axis-aligned elliptical bowl in (slope, height-at-mean) coordinates. A slope error costs more when is spread out, which is also why slopes are estimated more precisely from spread-out data. The figure below draws exactly this bowl.
- share of the variance of y explained by the line (training data)
- residual standard deviation; because two parameters were fitted
- standard error of the slope, for confidence intervals and t-tests
Why squares? Maximum likelihood. Assume with independent . The negative log-likelihood is , so for any the most likely line is the one that minimizes RSS. Least squares is MLE under Gaussian noise; with Laplace noise MLE would minimize absolute error instead. Putting a prior on turns this into Bayesian regression.
Gradient descent and its one hyperparameter. Without solving the equations, step downhill: . Along each eigenvector of the Hessian with eigenvalue , the error is multiplied by per step. It shrinks only if , and the slowest direction shrinks by , so the number of steps grows with the condition number . After standardizing (mean 0, standard deviation 1) the Hessian of a one-feature model becomes : every direction shrinks by , lands on the answer in one step, halves the error each step, and never converges.
Worked example: rent from floor area
Eight apartments in one neighbourhood. We fit monthly rent (in dollars) against area (m²) and then price a 75 m² apartment.
| Apt | Area x (m²) | Rent y ($) |
|---|---|---|
| 1 | 35 | 760 |
| 2 | 45 | 830 |
| 3 | 50 | 950 |
| 4 | 60 | 1010 |
| 5 | 70 | 1150 |
| 6 | 80 | 1220 |
| 7 | 90 | 1390 |
| 8 | 105 | 1520 |
| Apt | x − x̄ | y − ȳ | (x − x̄)(y − ȳ) | (x − x̄)² |
|---|---|---|---|---|
| 1 | −31.875 | −343.75 | 10957.03 | 1016.02 |
| 2 | −21.875 | −273.75 | 5988.28 | 478.52 |
| 3 | −16.875 | −153.75 | 2594.53 | 284.77 |
| 4 | −6.875 | −93.75 | 644.53 | 47.27 |
| 5 | 3.125 | 46.25 | 144.53 | 9.77 |
| 6 | 13.125 | 116.25 | 1525.78 | 172.27 |
| 7 | 23.125 | 286.25 | 6619.53 | 534.77 |
| 8 | 38.125 | 416.25 | 15869.53 | 1453.52 |
| Σ | 0 | 0 | 44343.75 | 3996.88 |
Fitting by hand
- 1Means. m², dollars.
- 2Sums of deviations (table above): , .
- 3Slope. : each extra m² adds about 11.09 dollars to the monthly rent.
- 4Intercept. . The fitted line is .
- 5Residuals (table below) sum to 0, as the first normal equation demands, and too. , so and : the line is typically about 23 dollars off.
- 6Fit quality. , so : area explains 99.1% of the variance in rent on this sample.
- 7Uncertainty of the slope. , , (). The slope is clearly not zero.
- 8Predict. A 75 m² apartment: dollars. The 95% prediction interval for a single new apartment is about 1123 to 1265 dollars.
| Area x | Rent y | Fitted ŷ | Residual e = y − ŷ | e² |
|---|---|---|---|---|
| 35 | 760 | 750.11 | 9.89 | 97.82 |
| 45 | 830 | 861.06 | −31.06 | 964.44 |
| 50 | 950 | 916.53 | 33.47 | 1120.34 |
| 60 | 1010 | 1027.47 | −17.47 | 305.36 |
| 70 | 1150 | 1138.42 | 11.58 | 134.08 |
| 80 | 1220 | 1249.37 | −29.37 | 862.40 |
| 90 | 1390 | 1360.31 | 29.69 | 881.33 |
| 105 | 1520 | 1526.73 | −6.73 | 45.32 |
See it: squares, the bowl, and the descent
Notice that the bowl's contours are axis-aligned ellipses, wider along the slope axis than you might expect: that is the term at work. Dragging the line parallel to itself only moves the dot vertically (height error); rotating it about only moves it horizontally (slope error). The two kinds of error add without interacting.
The algorithm, step by step
Fit(X, y) by ordinary least squares
- 1Build the design matrix. Prepend a column of ones to the feature matrix (skip it only if you truly want the line forced through the origin).
- 2Solve the least-squares problem with a QR decomposition : then is a triangular system. This is what "np.linalg.lstsq" (SVD) and scikit-learn (SciPy "lstsq") do. Avoid forming and inverting explicitly.
- 3Compute residuals , then RSS, MSE, and, if you need inference, .
- 4Check diagnostics: plot residuals against fitted values and against each feature, look for curves, funnels and isolated large residuals.
- 5Predict for new inputs, ideally only inside the range of the training features.
- 1w ← 0, b ← 0
- 2repeat:
- 3 ŷᵢ ← w·xᵢ + b
- 4 rᵢ ← ŷᵢ − yᵢ
- 5 w ← w − η·(2/n)Σ rᵢxᵢ
- 6 b ← b − η·(2/n)Σ rᵢ
- 7until MSE stops falling
Start with w = 0, b = 0: a flat line at ŷ = 0. MSE = 0.3010.
Cost
- Normal equations
- form in , solve in ; memory , so it can stream over rows
- QR / SVD (lstsq)
- with a larger constant, numerically stable; the default
- Gradient descent
- per full-batch step; number of steps grows with the condition number
- SGD / mini-batch
- per example; scales to data that does not fit in memory
- Prediction
- per input; the model is numbers
Variants and practical choices
Plain least squares has no hyperparameters to tune, so the practical decisions are about which solver, which loss and which features. Closely related models get their own articles: adding several features is multiple linear regression, adding powers of is polynomial regression, adding a penalty or is ridge or lasso regression, a full posterior over is Bayesian regression, and a non-parametric local average is KNN regression.
| Variant | Minimizes | Use when |
|---|---|---|
| OLS (lstsq / QR) | default; small to medium n, any d that fits in memory | |
| Normal equations | very large n, small d: accumulate and in one pass | |
| Gradient descent / SGD | iteratively | huge or streaming data, or as part of a larger model; standardize first |
| Weighted least squares | noise variance differs between points (heteroscedasticity) and you know the weights | |
| Huber regression | squared loss for small e, absolute for large | a few outliers you cannot remove; scikit-learn HuberRegressor |
| Theil–Sen / RANSAC | median of pairwise slopes / consensus of inliers | many gross outliers |
| Through the origin | , | physics says y = 0 at x = 0 (fit_intercept=False) |
| Assumption | Symptom when violated | Remedy |
|---|---|---|
| Linearity of the mean | curved band in residuals vs fitted | transform x or y (log), add polynomial terms, or use a nonlinear model |
| Independent errors | residuals correlated in time or by group | time-series models, clustered standard errors |
| Constant variance | funnel-shaped residual plot | log-transform y, weighted least squares, robust (HC) standard errors |
| Roughly Gaussian errors | skewed or heavy-tailed residual histogram | matters mostly for small-n intervals; bootstrap them, or use a robust loss |
| No dominant outliers | one point with a huge residual or leverage | inspect it; Huber / RANSAC; report with and without it |
In code
1import numpy as np23class LinearRegressionOLS:4 """Least squares via the normal equations, solved stably with lstsq (QR/SVD)."""5 def fit(self, X, y):6 X = np.asarray(X, float).reshape(len(y), -1)7 A = np.c_[np.ones(len(X)), X] # column of ones absorbs the bias8 theta, *_ = np.linalg.lstsq(A, y, rcond=None)9 self.intercept_, self.coef_ = theta[0], theta[1:]10 return self1112 def predict(self, X):13 X = np.asarray(X, float).reshape(-1, len(self.coef_))14 return X @ self.coef_ + self.intercept_1516def fit_simple(x, y):17 """The textbook one-feature formulas: w = Sxy / Sxx, b = ȳ − w·x̄."""18 dx, dy = x - x.mean(), y - y.mean()19 w = (dx * dy).sum() / (dx ** 2).sum()20 return w, y.mean() - w * x.mean()2122def fit_gd(x, y, lr=0.25, steps=50):23 """Gradient descent on standardized x (well-conditioned), then undo the scaling."""24 mu, sd = x.mean(), x.std()25 z = (x - mu) / sd26 c, s = 0.0, 0.0 # ŷ = c + s·z27 for _ in range(steps):28 r = c + s * z - y # residuals29 c -= lr * 2 * r.mean()30 s -= lr * 2 * (r * z).mean()31 w = s / sd32 return w, c - w * mu3334area = np.array([35, 45, 50, 60, 70, 80, 90, 105.])35rent = np.array([760, 830, 950, 1010, 1150, 1220, 1390, 1520.])3637w, b = fit_simple(area, rent)38print(f"closed form : w = {w:.4f}, b = {b:.2f}")39m = LinearRegressionOLS().fit(area, rent)40print(f"lstsq : w = {m.coef_[0]:.4f}, b = {m.intercept_:.2f}")41print(f"grad descent: w = {fit_gd(area, rent)[0]:.4f}, b = {fit_gd(area, rent)[1]:.2f}")42res = rent - m.predict(area)43print(f"MSE = {np.mean(res**2):.2f}, R² = {1 - res @ res / ((rent - rent.mean())**2).sum():.4f}")44print(f"rent for 75 m²: {m.predict([75])[0]:.2f}")4546# closed form : w = 11.0946, b = 361.8047# lstsq : w = 11.0946, b = 361.8048# grad descent: w = 11.0946, b = 361.8049# MSE = 551.39, R² = 0.991150# rent for 75 m²: 1193.891import numpy as np2from scipy import stats3from sklearn.linear_model import LinearRegression4from sklearn.metrics import mean_squared_error, r2_score56area = np.array([35, 45, 50, 60, 70, 80, 90, 105.]).reshape(-1, 1)7rent = np.array([760, 830, 950, 1010, 1150, 1220, 1390, 1520.])89model = LinearRegression().fit(area, rent)10pred = model.predict(area)11print(f"w = {model.coef_[0]:.4f}, b = {model.intercept_:.2f}")12print(f"MSE = {mean_squared_error(rent, pred):.2f}, "13 f"RMSE = {mean_squared_error(rent, pred) ** 0.5:.2f}, R² = {r2_score(rent, pred):.4f}")14print(f"rent for 75 m²: {model.predict([[75]])[0]:.2f}")1516# Inference: standard error of the slope and a 95% prediction interval17fit = stats.linregress(area.ravel(), rent)18print(f"SE(w) = {fit.stderr:.4f}, t = {fit.slope / fit.stderr:.2f}, p = {fit.pvalue:.1e}")19x, n = area.ravel(), len(rent)20s = np.sqrt(((rent - pred) ** 2).sum() / (n - 2)) # residual std, n − 2 dof21se_new = s * np.sqrt(1 + 1 / n + (75 - x.mean()) ** 2 / ((x - x.mean()) ** 2).sum())22t = stats.t.ppf(0.975, n - 2)23y75 = model.predict([[75]])[0]24print(f"95% prediction interval at 75 m²: [{y75 - t * se_new:.0f}, {y75 + t * se_new:.0f}]")2526# w = 11.0946, b = 361.8027# MSE = 551.39, RMSE = 23.48, R² = 0.991128# rent for 75 m²: 1193.8929# SE(w) = 0.4289, t = 25.87, p = 2.2e-0730# 95% prediction interval at 75 m²: [1123, 1265]Use when
- +You need a fast, transparent baseline: the slope is "dollars per m²", and the whole model fits on a sticky note.
- +The relationship is close to linear, or becomes linear after a transform such as .
- +You need uncertainty and inference: standard errors, confidence and prediction intervals come almost for free.
- +Data is small: two parameters rarely overfit, and the exact solution needs no tuning.
Avoid when
- −The response curves, saturates or has thresholds: residuals will show structure. Use polynomial terms, splines, trees or boosting.
- −There are gross outliers or heavy-tailed noise: squared error lets a few points dominate. Use Huber, Theil–Sen or RANSAC.
- −You must predict far outside the training range: a straight line extrapolates without limit.
- −The target is a category or a probability: use logistic regression, not a line clipped to [0, 1].
Where it's used
Pricing and valuation
Rent or house price per m², used-car price per year of age: the slope is the quoted rate.
Calibration in the lab
Instrument readings against known standards (e.g. absorbance vs concentration, Beer–Lambert) and inverting the line to measure unknowns.
Economics and finance
A stock's beta is the least-squares slope of its returns on the market's returns.
Trend and baseline estimation
Linear trend of temperature or sales over time; the first baseline that any fancier model must beat.
Check yourself
For a least-squares fit that includes an intercept, the residuals always…
Check yourself
In the rent example m² and . What does the fitted line predict for a 66.875 m² apartment?
Check yourself
You re-express area in units of 10 m² (so every x is divided by 10) and refit. The slope becomes…
Check yourself
Gradient descent on one standardized feature, loss MSE, Hessian . Which learning rate fails to converge?
Remember
- ✓Model , loss = mean squared error; the loss is a convex bowl with one minimum.
- ✓Zero gradient gives , ; in matrix form , so . Solve it with QR / lstsq, not an explicit inverse.
- ✓Residuals sum to zero, are uncorrelated with , and the line passes through : the fit is an orthogonal projection.
- ✓Least squares is maximum likelihood under Gaussian noise; , and quantify fit and uncertainty.
- ✓Gradient descent needs standardized features and a learning rate below 2/λmax; the closed form needs neither.
- ✓Squared error is outlier-sensitive and lines extrapolate blindly: plot residuals, and use robust losses or nonlinear models when they show structure.