# Projections & Orthogonality

> The closest point to b in a subspace is its orthogonal projection — the residual is perpendicular to the subspace — which is least squares.

Source: https://www.mlfoundry.in/learn/projections-orthogonality  
Section: Mathematics › Linear Algebra  
Level: core · 6 min read · updated 2026-10-03

**The orthogonal projection:** of a vector $b$ onto a subspace $S$ is the unique point $p\in S$ whose **residual** $e=b-p$ is orthogonal to every vector in $S$. It is also the point of $S$ closest to $b$.

- **Notation:** $p=Pb$, $P$ the projection matrix; $u\perp v$ ⇔ $u^\top v=0$
- **Input → output:** a vector $b\in\mathbb{R}^n$ and a subspace (spanned by the columns of $A$) → the closest point $p$ and residual $e$
- **Formula:** $P=A(A^\top A)^{-1}A^\top$; onto a line: $p=\dfrac{a^\top b}{a^\top a}\,a$
- **Key property:** $A^\top(b-p)=0$, $P^2=P=P^\top$, $\lVert b\rVert^2=\lVert p\rVert^2+\lVert e\rVert^2$
- **Prerequisites:** dot product, matrix multiplication, linear systems & rank
- **Used in:** linear regression, PCA, QR decomposition, partialling out confounders

## The intuition

Shine a light straight down onto a floor: the shadow of a pole's tip is the point on the floor closest to the tip, and the line from tip to shadow hits the floor at a right angle. Any other floor point is farther away, because you would be walking the hypotenuse of a right triangle instead of a leg.

Least squares is that picture in $\mathbb{R}^n$. Your target vector $y$ holds one number per sample. Every prediction a linear model can make, $X\beta$, lies in the **column space** of $X$ — a flat subspace. The best prediction is the shadow of $y$ on that subspace, and the residuals are the perpendicular drop: orthogonal to every feature column.

## The math

**Onto a line.** Write $p=c\,a$ for a nonzero direction $a$. Orthogonality of the residual, $a^\top(b-c\,a)=0$, gives $c=a^\top b/a^\top a$:

$$
p=\frac{a^\top b}{a^\top a}\,a=\underbrace{\frac{aa^\top}{a^\top a}}_{P}\,b,\qquad e=b-p,\qquad a^\top e=0
$$

- $a$: direction spanning the line
- $P$: $n\times n$ projection matrix, rank 1
- $e$: residual, perpendicular to the line

**Onto a subspace.** Let the subspace be the column space of an $n\times k$ matrix $A$ with independent columns, so every point in it is $A\hat x$ for some coefficients $\hat x\in\mathbb{R}^k$. Requiring the residual to be orthogonal to each column, $A^\top(b-A\hat x)=0$, gives the **normal equations**:

$$
A^\top A\,\hat x=A^\top b\ \Rightarrow\ \hat x=(A^\top A)^{-1}A^\top b,\qquad P=A(A^\top A)^{-1}A^\top
$$

- $\hat x$: coefficients of the projection in the columns of A — the least-squares solution
- $A^\top A$: $k\times k$ Gram matrix, invertible because the columns are independent

**Why it is the closest point.** For any $q$ in the subspace, $b-q=e+(p-q)$, and $p-q$ lies in the subspace, so it is orthogonal to $e$. Pythagoras gives $\lVert b-q\rVert^2=\lVert e\rVert^2+\lVert p-q\rVert^2\ge\lVert e\rVert^2$, with equality only at $q=p$. Minimizing $\lVert b-Ax\rVert^2$ and solving $A^\top(b-Ax)=0$ are the same problem.

**Properties of $P$.** Projecting twice changes nothing ($P^2=P$), $P$ is symmetric, its eigenvalues are 1 (on the subspace) and 0 (on its orthogonal complement), and $\operatorname{tr}P=k$. The complement $I-P$ is also a projection, onto everything perpendicular to the subspace — it maps $b$ to $e$.

**Orthonormal bases make it trivial.** If the columns of $Q$ are orthonormal ($Q^\top Q=I$), the Gram matrix disappears: $P=QQ^\top$ and $\hat x=Q^\top b$, so each coefficient is a single dot product. Gram–Schmidt — and its stable matrix form, the QR decomposition $A=QR$ — converts any basis into an orthonormal one.

## See it

> Figure: Line mode: a = (4, 1), b = (1.5, 3) gives a·b = 9, p = (2.12, 0.53) and ‖e‖ = 2.547, with e·a = 0 wherever you drag. Drag along span(a) to place another point q — ‖b − q‖ is never below ‖e‖. Least-squares mode: six points, minimum SSE = 1.092 where 1ᵀr = xᵀr = 0; nudging the slope by +0.30 raises SSE to 6.944 and makes the residuals stop being orthogonal to the columns (xᵀr = −19.506).

## Worked example

**A line.** $a=(2,1)$, $b=(1,3)$: $a^\top b=5$, $a^\top a=5$, so $p=(2,1)$ and $e=(-1,2)$. Check: $a^\top e=-2+2=0$, and $\lVert b\rVert^2=10=\lVert p\rVert^2+\lVert e\rVert^2=5+5$.

**A 3-point regression.** Fit $y=\beta_0+\beta_1x$ to $(0,1),(1,2),(2,4)$. The design matrix has an intercept column and a feature column, $X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}$, with $y=(1,2,4)$. Then $X^\top X=\begin{pmatrix}3&3\\3&5\end{pmatrix}$ and $X^\top y=(7,10)$; solving gives $\hat\beta=(5/6,\ 3/2)$.

Fitted values $\hat y=X\hat\beta=(0.833,\ 2.333,\ 3.833)$ — the projection of $y$ onto the column space. Residuals $e=(1/6,\ -1/3,\ 1/6)$: they sum to 0 (orthogonal to the intercept column) and $0\cdot\tfrac16+1\cdot(-\tfrac13)+2\cdot\tfrac16=0$ (orthogonal to $x$). $\lVert e\rVert^2=1/6\approx0.167$, and indeed $\lVert y\rVert^2=21=20.833+0.167$.

## In code

```python
import numpy as np

a, b = np.array([2.0, 1.0]), np.array([1.0, 3.0])
p = (a @ b) / (a @ a) * a                  # array([2., 1.])
e = b - p                                  # array([-1., 2.]);  a @ e == 0.0

X = np.array([[1, 0], [1, 1], [1, 2]], dtype=float)   # intercept + feature
y = np.array([1.0, 2.0, 4.0])
beta = np.linalg.solve(X.T @ X, X.T @ y)   # array([0.8333, 1.5])
y_hat = X @ beta                           # array([0.8333, 2.3333, 3.8333])
print(X.T @ (y - y_hat))                   # ≈ [0, 0]: residual ⟂ every column

beta2, *_ = np.linalg.lstsq(X, y, rcond=None)   # same β, numerically safer
Q, R = np.linalg.qr(X)                     # orthonormal basis of col(X)
P = Q @ Q.T                                # projection matrix
print(np.allclose(P @ P, P), np.allclose(P @ y, y_hat))   # True True
```

Projection onto a line, then least squares three ways — normal equations, lstsq and QR — all giving the same projection.

- **Linear regression:** OLS fitted values are $Py$ with $P=X(X^\top X)^{-1}X^\top$ (the *hat matrix*); its diagonal gives each point's leverage.
- **PCA:** Each sample is projected onto the span of the top-$k$ principal directions; the reconstruction error is the orthogonal residual.
- **Partialling out:** To remove a confounder $z$, replace $y$ and $x$ by their residuals $(I-P_z)y$, $(I-P_z)x$; regressing one on the other recovers the multiple-regression coefficient (Frisch–Waugh–Lovell).
- **Embedding debiasing:** Subtracting $\operatorname{proj}_g x$ removes a learned direction $g$ (e.g. a gender axis) from every embedding, leaving $x\perp g$.

> **Adding per-column shadows only works for orthogonal columns:** Projecting $b$ onto each column separately and summing gives the true projection only when the columns are orthogonal. With correlated features the separate shadows double-count shared directions — exactly why a multiple-regression coefficient differs from the simple-regression slope of the same feature.

> **Don't form the normal equations for real work:** Solving $X^\top X\beta=X^\top y$ squares the condition number: $\kappa(X^\top X)=\kappa(X)^2$. With $\kappa(X)=10^6$, double precision is left with almost no correct digits. `np.linalg.lstsq` (SVD) or a QR solve projects without ever forming $X^\top X$.

> **R² is a squared cosine:** After centering, $\hat y$ is the projection of $y$, so $R^2=\lVert\hat y\rVert^2/\lVert y\rVert^2=\cos^2\theta$, where $\theta$ is the angle between $y$ and the column space. A perfect fit means $y$ already lies in the subspace.

**Question:** Project $b=(3,1)$ onto the line spanned by $a=(1,1)$. What is $p$?

- A. (3, 0)
- B. (2, 2)
- C. (1, 1)
- D. (4, 4)

**Answer:** B. (2, 2) — $a^\top b=4$, $a^\top a=2$, so $p=\tfrac42(1,1)=(2,2)$. Check: $e=(1,-1)$ and $a^\top e=0$.

**Question:** An OLS model includes an intercept. What do its training residuals sum to?

- A. Exactly 0
- B. The sample mean of y
- C. The SSE
- D. It depends on the data

**Answer:** A. Exactly 0 — The intercept column is $\mathbf 1$, and the residual is orthogonal to every column: $\mathbf 1^\top e=\sum_i e_i=0$.

**Key points**

- The projection $p$ is the closest point of a subspace to $b$; the residual $b-p$ is orthogonal to the whole subspace.
- Orthogonality to the columns, $A^\top(b-A\hat x)=0$, *is* the least-squares condition (the normal equations).
- $P=A(A^\top A)^{-1}A^\top$ is symmetric and idempotent; with orthonormal $Q$ it is simply $QQ^\top$.
- In practice project via QR or SVD (lstsq), never by inverting the Gram matrix.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
