Definition
The orthogonal projection of a vector onto a subspace is the unique point whose residual is orthogonal to every vector in . It is also the point of closest to .
At a glance
- Notation
- , the projection matrix; ⇔
- Input → output
- a vector and a subspace (spanned by the columns of ) → the closest point and residual
- Formula
- ; onto a line:
- Key property
- , ,
- Prerequisites
- dot product, matrix multiplication, linear systems & rank
- Used in
- linear regression, PCA, QR decomposition, partialling out confounders
The intuition
Shine a light straight down onto a floor: the shadow of a pole's tip is the point on the floor closest to the tip, and the line from tip to shadow hits the floor at a right angle. Any other floor point is farther away, because you would be walking the hypotenuse of a right triangle instead of a leg.
Least squares is that picture in . Your target vector holds one number per sample. Every prediction a linear model can make, , lies in the column space of — a flat subspace. The best prediction is the shadow of on that subspace, and the residuals are the perpendicular drop: orthogonal to every feature column.
The math
Onto a line. Write for a nonzero direction . Orthogonality of the residual, , gives :
- direction spanning the line
- projection matrix, rank 1
- residual, perpendicular to the line
Onto a subspace. Let the subspace be the column space of an matrix with independent columns, so every point in it is for some coefficients . Requiring the residual to be orthogonal to each column, , gives the normal equations:
- coefficients of the projection in the columns of A — the least-squares solution
- Gram matrix, invertible because the columns are independent
Why it is the closest point. For any in the subspace, , and lies in the subspace, so it is orthogonal to . Pythagoras gives , with equality only at . Minimizing and solving are the same problem.
Properties of . Projecting twice changes nothing (), is symmetric, its eigenvalues are 1 (on the subspace) and 0 (on its orthogonal complement), and . The complement is also a projection, onto everything perpendicular to the subspace — it maps to .
Orthonormal bases make it trivial. If the columns of are orthonormal (), the Gram matrix disappears: and , so each coefficient is a single dot product. Gram–Schmidt — and its stable matrix form, the QR decomposition — converts any basis into an orthonormal one.
See it
Worked example
A line. , : , , so and . Check: , and .
A 3-point regression. Fit to . The design matrix has an intercept column and a feature column, , with . Then and ; solving gives .
Fitted values — the projection of onto the column space. Residuals : they sum to 0 (orthogonal to the intercept column) and (orthogonal to ). , and indeed .
In code
1import numpy as np23a, b = np.array([2.0, 1.0]), np.array([1.0, 3.0])4p = (a @ b) / (a @ a) * a # array([2., 1.])5e = b - p # array([-1., 2.]); a @ e == 0.067X = np.array([[1, 0], [1, 1], [1, 2]], dtype=float) # intercept + feature8y = np.array([1.0, 2.0, 4.0])9beta = np.linalg.solve(X.T @ X, X.T @ y) # array([0.8333, 1.5])10y_hat = X @ beta # array([0.8333, 2.3333, 3.8333])11print(X.T @ (y - y_hat)) # ≈ [0, 0]: residual ⟂ every column1213beta2, *_ = np.linalg.lstsq(X, y, rcond=None) # same β, numerically safer14Q, R = np.linalg.qr(X) # orthonormal basis of col(X)15P = Q @ Q.T # projection matrix16print(np.allclose(P @ P, P), np.allclose(P @ y, y_hat)) # True TrueWhere it's used
Linear regression
OLS fitted values are with (the hat matrix); its diagonal gives each point's leverage.
PCA
Each sample is projected onto the span of the top- principal directions; the reconstruction error is the orthogonal residual.
Partialling out
To remove a confounder , replace and by their residuals , ; regressing one on the other recovers the multiple-regression coefficient (Frisch–Waugh–Lovell).
Embedding debiasing
Subtracting removes a learned direction (e.g. a gender axis) from every embedding, leaving .
Check yourself
Project onto the line spanned by . What is ?
Check yourself
An OLS model includes an intercept. What do its training residuals sum to?
Remember
- ✓The projection is the closest point of a subspace to ; the residual is orthogonal to the whole subspace.
- ✓Orthogonality to the columns, , is the least-squares condition (the normal equations).
- ✓ is symmetric and idempotent; with orthonormal it is simply .
- ✓In practice project via QR or SVD (lstsq), never by inverting the Gram matrix.