Definition
A norm is a function that gives every vector a non-negative length, is zero only for the zero vector, scales with the vector () and obeys the triangle inequality . The distance between two points is the norm of their difference, .
At a glance
- Notation
- for the norm; plain usually means
- Input → output
- a vector → a non-negative number
- The common three
- = sum of absolute values · = Euclidean length · = largest absolute coordinate
- Key property
- for every
- Prerequisites
- vectors & the dot product
- Used in
- k-NN, k-means, lasso and ridge, adversarial robustness, gradient clipping
The intuition
Length sounds unambiguous until you ask how to combine coordinates. A taxi on a street grid travels (). A bird flies (). A chess king needs moves (). Same two points, three valid distances — and every algorithm built on distance inherits whichever one you pick.
The clearest picture of a norm is its unit ball : a diamond for , a disc for , a square for . Every property of the norm is visible in that shape.
The math
- order of the norm,
- number of coordinates
Only comes from a dot product, which is why it alone has angles and orthogonal projections. Below the formula breaks the triangle inequality — the "ball" stops being convex — so it is not a norm.
Why the shape matters for regularization. Minimizing a loss subject to means finding where the loss contours first touch a diamond. That usually happens at a corner, where some coordinates are exactly zero. This is why the lasso () produces sparse weights, while ridge (, a round ball) only shrinks them.
See it
Drag a and b. staircase = L1 path, straight = L2, bar = L∞ (largest gap). Dotted: all points at the same Lp distance from a as b.
Worked example
: , , , and in between — the ordering holds. Normalizing for gives the unit vector .
The choice changes answers. From the origin, and are at distances 4 and 3, distances and 3, distances 2 and 3. The nearest neighbour is under and under the other two.
In code
1import numpy as np2from scipy.spatial.distance import cdist34x = np.array([3.0, -4.0])5np.linalg.norm(x, 1), np.linalg.norm(x), np.linalg.norm(x, np.inf) # (7.0, 5.0, 4.0)67origin = np.array([[0, 0]]); pts = np.array([[2, 2], [3, 0]])8cdist(origin, pts, "cityblock") # [[4., 3.]] L19cdist(origin, pts, "euclidean") # [[2.83, 3.]] L210cdist(origin, pts, "chebyshev") # [[2., 3.]] L∞Where it's used
Nearest neighbours & clustering
The metric decides who is close; L1 is less dominated by a single large coordinate gap than L2.
Regularization
Lasso (L1) zeroes weights, ridge (L2) shrinks them, elastic net mixes both.
Adversarial robustness
Attacks are bounded in (each pixel changes by at most ) or in .
Training stability
Gradient clipping rescales any gradient whose L2 norm exceeds a threshold; embeddings are L2-normalized before cosine search.
Check yourself
For , what are , and ?
Check yourself
Why does L1 regularization produce weights that are exactly zero, while L2 does not?
Remember
- ✓: sum, Euclidean, max.
- ✓Unit balls: diamond, disc, square — the shape is the norm.
- ✓Only L2 has angles and projections; L1's corners create sparsity.
- ✓Pick the norm deliberately, and standardize features first.