Projections & Orthogonality

The closest point to b in a subspace is its orthogonal projection — the residual is perpendicular to the subspace — which is least squares.

6 min read628 wordscore5 viewsupdated 2026-10-03

Definition

The orthogonal projection of a vector bb onto a subspace SS is the unique point p∈Sp\in S whose residual e=b−pe=b-p is orthogonal to every vector in SS. It is also the point of SS closest to bb.

At a glance

Notation
p=Pbp=Pb, PP the projection matrix; u⊥vu\perp v ⇔ u⊤v=0u^\top v=0
Input → output
a vector b∈Rnb\in\mathbb{R}^n and a subspace (spanned by the columns of AA) → the closest point pp and residual ee
Formula
P=A(A⊤A)−1A⊤P=A(A^\top A)^{-1}A^\top; onto a line: p=a⊤ba⊤a ap=\dfrac{a^\top b}{a^\top a}\,a
Key property
A⊤(b−p)=0A^\top(b-p)=0, P2=P=P⊤P^2=P=P^\top, ∥b∥2=∥p∥2+∥e∥2\lVert b\rVert^2=\lVert p\rVert^2+\lVert e\rVert^2
Prerequisites
dot product, matrix multiplication, linear systems & rank
Used in
linear regression, PCA, QR decomposition, partialling out confounders

The intuition

Shine a light straight down onto a floor: the shadow of a pole's tip is the point on the floor closest to the tip, and the line from tip to shadow hits the floor at a right angle. Any other floor point is farther away, because you would be walking the hypotenuse of a right triangle instead of a leg.

Least squares is that picture in Rn\mathbb{R}^n. Your target vector yy holds one number per sample. Every prediction a linear model can make, XβX\beta, lies in the column space of XX — a flat subspace. The best prediction is the shadow of yy on that subspace, and the residuals are the perpendicular drop: orthogonal to every feature column.

The math

Onto a line. Write p=c ap=c\,a for a nonzero direction aa. Orthogonality of the residual, a⊤(b−c a)=0a^\top(b-c\,a)=0, gives c=a⊤b/a⊤ac=a^\top b/a^\top a:

p=a⊤ba⊤a a=aa⊤a⊤a⏟P b,e=b−p,a⊤e=0p=\frac{a^\top b}{a^\top a}\,a=\underbrace{\frac{aa^\top}{a^\top a}}_{P}\,b,\qquad e=b-p,\qquad a^\top e=0
aa
direction spanning the line
PP
n×nn\times n projection matrix, rank 1
ee
residual, perpendicular to the line

Onto a subspace. Let the subspace be the column space of an n×kn\times k matrix AA with independent columns, so every point in it is Ax^A\hat x for some coefficients x^∈Rk\hat x\in\mathbb{R}^k. Requiring the residual to be orthogonal to each column, A⊤(b−Ax^)=0A^\top(b-A\hat x)=0, gives the normal equations:

A⊤A x^=A⊤b ⇒ x^=(A⊤A)−1A⊤b,P=A(A⊤A)−1A⊤A^\top A\,\hat x=A^\top b\ \Rightarrow\ \hat x=(A^\top A)^{-1}A^\top b,\qquad P=A(A^\top A)^{-1}A^\top
x^\hat x
coefficients of the projection in the columns of A — the least-squares solution
A⊤AA^\top A
k×kk\times k Gram matrix, invertible because the columns are independent

Why it is the closest point. For any qq in the subspace, b−q=e+(p−q)b-q=e+(p-q), and p−qp-q lies in the subspace, so it is orthogonal to ee. Pythagoras gives ∥b−q∥2=∥e∥2+∥p−q∥2≥∥e∥2\lVert b-q\rVert^2=\lVert e\rVert^2+\lVert p-q\rVert^2\ge\lVert e\rVert^2, with equality only at q=pq=p. Minimizing ∥b−Ax∥2\lVert b-Ax\rVert^2 and solving A⊤(b−Ax)=0A^\top(b-Ax)=0 are the same problem.

Properties of PP. Projecting twice changes nothing (P2=PP^2=P), PP is symmetric, its eigenvalues are 1 (on the subspace) and 0 (on its orthogonal complement), and tr⁡P=k\operatorname{tr}P=k. The complement I−PI-P is also a projection, onto everything perpendicular to the subspace — it maps bb to ee.

Orthonormal bases make it trivial. If the columns of QQ are orthonormal (Q⊤Q=IQ^\top Q=I), the Gram matrix disappears: P=QQ⊤P=QQ^\top and x^=Q⊤b\hat x=Q^\top b, so each coefficient is a single dot product. Gram–Schmidt — and its stable matrix form, the QR decomposition A=QRA=QR — converts any basis into an orthonormal one.

See it

drag a and b
span(a)apb
a·b = 9.00p = (2.12, 0.53)‖e‖ = 2.547e·a = 0.000drag along the dashed line to try another point q
fig.Line mode: a = (4, 1), b = (1.5, 3) gives a·b = 9, p = (2.12, 0.53) and ‖e‖ = 2.547, with e·a = 0 wherever you drag. Drag along span(a) to place another point q — ‖b − q‖ is never below ‖e‖. Least-squares mode: six points, minimum SSE = 1.092 where 1ᵀr = xᵀr = 0; nudging the slope by +0.30 raises SSE to 6.944 and makes the residuals stop being orthogonal to the columns (xᵀr = −19.506).

Worked example

A line. a=(2,1)a=(2,1), b=(1,3)b=(1,3): a⊤b=5a^\top b=5, a⊤a=5a^\top a=5, so p=(2,1)p=(2,1) and e=(−1,2)e=(-1,2). Check: a⊤e=−2+2=0a^\top e=-2+2=0, and ∥b∥2=10=∥p∥2+∥e∥2=5+5\lVert b\rVert^2=10=\lVert p\rVert^2+\lVert e\rVert^2=5+5.

A 3-point regression. Fit y=β0+β1xy=\beta_0+\beta_1x to (0,1),(1,2),(2,4)(0,1),(1,2),(2,4). The design matrix has an intercept column and a feature column, X=(101112)X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}, with y=(1,2,4)y=(1,2,4). Then X⊤X=(3335)X^\top X=\begin{pmatrix}3&3\\3&5\end{pmatrix} and X⊤y=(7,10)X^\top y=(7,10); solving gives β^=(5/6, 3/2)\hat\beta=(5/6,\ 3/2).

Fitted values y^=Xβ^=(0.833, 2.333, 3.833)\hat y=X\hat\beta=(0.833,\ 2.333,\ 3.833) — the projection of yy onto the column space. Residuals e=(1/6, −1/3, 1/6)e=(1/6,\ -1/3,\ 1/6): they sum to 0 (orthogonal to the intercept column) and 0⋅16+1⋅(−13)+2⋅16=00\cdot\tfrac16+1\cdot(-\tfrac13)+2\cdot\tfrac16=0 (orthogonal to xx). ∥e∥2=1/6≈0.167\lVert e\rVert^2=1/6\approx0.167, and indeed ∥y∥2=21=20.833+0.167\lVert y\rVert^2=21=20.833+0.167.

In code

python
1import numpy as np
2
3a, b = np.array([2.0, 1.0]), np.array([1.0, 3.0])
4p = (a @ b) / (a @ a) * a # array([2., 1.])
5e = b - p # array([-1., 2.]); a @ e == 0.0
6
7X = np.array([[1, 0], [1, 1], [1, 2]], dtype=float) # intercept + feature
8y = np.array([1.0, 2.0, 4.0])
9beta = np.linalg.solve(X.T @ X, X.T @ y) # array([0.8333, 1.5])
10y_hat = X @ beta # array([0.8333, 2.3333, 3.8333])
11print(X.T @ (y - y_hat)) # ≈ [0, 0]: residual ⟂ every column
12
13beta2, *_ = np.linalg.lstsq(X, y, rcond=None) # same β, numerically safer
14Q, R = np.linalg.qr(X) # orthonormal basis of col(X)
15P = Q @ Q.T # projection matrix
16print(np.allclose(P @ P, P), np.allclose(P @ y, y_hat)) # True True
Projection onto a line, then least squares three ways — normal equations, lstsq and QR — all giving the same projection.

Where it's used

Linear regression

OLS fitted values are PyPy with P=X(X⊤X)−1X⊤P=X(X^\top X)^{-1}X^\top (the hat matrix); its diagonal gives each point's leverage.

PCA

Each sample is projected onto the span of the top-kk principal directions; the reconstruction error is the orthogonal residual.

Partialling out

To remove a confounder zz, replace yy and xx by their residuals (I−Pz)y(I-P_z)y, (I−Pz)x(I-P_z)x; regressing one on the other recovers the multiple-regression coefficient (Frisch–Waugh–Lovell).

Embedding debiasing

Subtracting proj⁡gx\operatorname{proj}_g x removes a learned direction gg (e.g. a gender axis) from every embedding, leaving x⊥gx\perp g.

Check yourself

Project b=(3,1)b=(3,1) onto the line spanned by a=(1,1)a=(1,1). What is pp?

Check yourself

An OLS model includes an intercept. What do its training residuals sum to?

Remember

  • ✓The projection pp is the closest point of a subspace to bb; the residual b−pb-p is orthogonal to the whole subspace.
  • ✓Orthogonality to the columns, A⊤(b−Ax^)=0A^\top(b-A\hat x)=0, is the least-squares condition (the normal equations).
  • ✓P=A(A⊤A)−1A⊤P=A(A^\top A)^{-1}A^\top is symmetric and idempotent; with orthonormal QQ it is simply QQ⊤QQ^\top.
  • ✓In practice project via QR or SVD (lstsq), never by inverting the Gram matrix.
← back to Mathematics roadmap