# Norms & Distances

> A norm measures a vector's size; L1, L2 and L∞ measure it differently, and each choice changes what 'close' means.

Source: https://www.mlfoundry.in/learn/norms-distances  
Section: Mathematics › Linear Algebra  
Level: core · 5 min read · updated 2026-10-03

**A norm:** is a function $\lVert\cdot\rVert$ that gives every vector a non-negative length, is zero only for the zero vector, scales with the vector ($\lVert\alpha x\rVert=|\alpha|\,\lVert x\rVert$) and obeys the triangle inequality $\lVert x+y\rVert\le\lVert x\rVert+\lVert y\rVert$. The **distance** between two points is the norm of their difference, $d(x,y)=\lVert x-y\rVert$.

- **Notation:** $\lVert x\rVert_p$ for the $L_p$ norm; plain $\lVert x\rVert$ usually means $L_2$
- **Input → output:** a vector → a non-negative number
- **The common three:** $L_1$ = sum of absolute values · $L_2$ = Euclidean length · $L_\infty$ = largest absolute coordinate
- **Key property:** $\lVert x\rVert_\infty\le\lVert x\rVert_2\le\lVert x\rVert_1$ for every $x$
- **Prerequisites:** vectors & the dot product
- **Used in:** k-NN, k-means, lasso and ridge, adversarial robustness, gradient clipping

## The intuition

Length sounds unambiguous until you ask how to combine coordinates. A taxi on a street grid travels $|\Delta x|+|\Delta y|$ ($L_1$). A bird flies $\sqrt{\Delta x^2+\Delta y^2}$ ($L_2$). A chess king needs $\max(|\Delta x|,|\Delta y|)$ moves ($L_\infty$). Same two points, three valid distances — and every algorithm built on distance inherits whichever one you pick.

The clearest picture of a norm is its **unit ball** $\{x:\lVert x\rVert\le1\}$: a diamond for $L_1$, a disc for $L_2$, a square for $L_\infty$. Every property of the norm is visible in that shape.

## The math

$$
\lVert x\rVert_p=\Big(\sum_{i=1}^n|x_i|^p\Big)^{1/p}\ \ (p\ge1),\qquad \lVert x\rVert_\infty=\max_i|x_i|=\lim_{p\to\infty}\lVert x\rVert_p
$$

- $p$: order of the norm, $1\le p\le\infty$
- $n$: number of coordinates

$$
\lVert x\rVert_1=\sum_i|x_i|,\qquad \lVert x\rVert_2=\sqrt{\textstyle\sum_i x_i^2}=\sqrt{x\cdot x},\qquad \lVert x\rVert_\infty=\max_i|x_i|
$$

Only $L_2$ comes from a dot product, which is why it alone has angles and orthogonal projections. Below $p=1$ the formula breaks the triangle inequality — the "ball" stops being convex — so it is not a norm.

**Why the shape matters for regularization.** Minimizing a loss subject to $\lVert w\rVert_1\le t$ means finding where the loss contours first touch a diamond. That usually happens at a corner, where some coordinates are exactly zero. This is why the lasso ($L_1$) produces **sparse** weights, while ridge ($L_2$, a round ball) only shrinks them.

## See it

> Figure: Slide p from 1 to ∞: the unit ball morphs from diamond to disc to square. Drag the points — for a = (−1.5, −0.5), b = (1.5, 1.5): L1 = 5, L2 ≈ 3.61, L∞ = 3. The dotted curve is every point as far from a as b is.

## Worked example

$x=(3,-4)$: $\lVert x\rVert_1=7$, $\lVert x\rVert_2=5$, $\lVert x\rVert_\infty=4$, and in between $\lVert x\rVert_3=91^{1/3}\approx4.498$ — the ordering $\infty\le3\le2\le1$ holds. Normalizing for $L_2$ gives the unit vector $(0.6,-0.8)$.

The choice changes answers. From the origin, $p_1=(2,2)$ and $p_2=(3,0)$ are at $L_1$ distances 4 and 3, $L_2$ distances $2.83$ and 3, $L_\infty$ distances 2 and 3. The nearest neighbour is $p_2$ under $L_1$ and $p_1$ under the other two.

## In code

```python
import numpy as np
from scipy.spatial.distance import cdist

x = np.array([3.0, -4.0])
np.linalg.norm(x, 1), np.linalg.norm(x), np.linalg.norm(x, np.inf)   # (7.0, 5.0, 4.0)

origin = np.array([[0, 0]]); pts = np.array([[2, 2], [3, 0]])
cdist(origin, pts, "cityblock")   # [[4., 3.]]       L1
cdist(origin, pts, "euclidean")   # [[2.83, 3.]]     L2
cdist(origin, pts, "chebyshev")   # [[2., 3.]]       L∞
```

Norms of one vector, and pairwise distances under three metrics.

- **Nearest neighbours & clustering:** The metric decides who is close; L1 is less dominated by a single large coordinate gap than L2.
- **Regularization:** Lasso (L1) zeroes weights, ridge (L2) shrinks them, elastic net mixes both.
- **Adversarial robustness:** Attacks are bounded in $L_\infty$ (each pixel changes by at most $\varepsilon$) or in $L_2$.
- **Training stability:** Gradient clipping rescales any gradient whose L2 norm exceeds a threshold; embeddings are L2-normalized before cosine search.

> **Scale before measuring:** Every norm combines coordinates directly, so a feature measured in thousands swamps one measured in fractions. Standardize features before computing any distance.

> **Distances concentrate in high dimensions:** For random high-dimensional data, the farthest and nearest points from a query end up at almost the same distance, so the ranking of neighbours becomes fragile. Reduce dimension or use learned embeddings before relying on raw distances.

**Question:** For $x=(1,-2,2)$, what are $\lVert x\rVert_1$, $\lVert x\rVert_2$ and $\lVert x\rVert_\infty$?

- A. (5, 3, 2)
- B. (3, 3, 2)
- C. (5, 9, 2)
- D. (1, 3, 2)

**Answer:** A. (5, 3, 2) — $1+2+2=5$; $\sqrt{1+4+4}=3$; $\max(1,2,2)=2$.

**Question:** Why does L1 regularization produce weights that are exactly zero, while L2 does not?

- A. L1 is cheaper to compute
- B. The L1 ball has corners on the axes, where loss contours tend to touch it
- C. L2 cannot be minimized by gradient descent
- D. L1 punishes large weights more than L2

**Answer:** B. The L1 ball has corners on the axes, where loss contours tend to touch it — The constrained optimum is where a loss contour first touches the ball. A diamond's corners sit on the axes — points where other coordinates are zero. A disc has no corners, so the touching point is generically off-axis.

**Key points**

- $\lVert x\rVert_p=(\sum|x_i|^p)^{1/p}$: $L_1$ sum, $L_2$ Euclidean, $L_\infty$ max.
- Unit balls: diamond, disc, square — the shape is the norm.
- Only L2 has angles and projections; L1's corners create sparsity.
- Pick the norm deliberately, and standardize features first.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
