Definition
The gradient of a differentiable function at a point is the vector of its partial derivatives, . Each partial derivative is the rate of change of when only that one coordinate moves; together they give the rate of change in every direction.
At a glance
- Notation
- , in one variable; , , in several
- Input → output
- a scalar function of variables → a vector field in
- Key property
- : the gradient is the steepest-ascent direction, perpendicular to level sets, and its length is the steepest slope
- Prerequisites
- limits and slopes, vectors & the dot product
- Used in
- gradient descent and every optimizer built on it, backprop, saliency maps, sensitivity analysis
The intuition
In one variable the derivative is the slope of the tangent line: nudge by a tiny and changes by about . With variables you can nudge in infinitely many directions, but differentiability means the response is linear in the nudge — so numbers, one slope per axis, determine all of them. Those numbers are the gradient.
Picture as terrain and its level sets (contour lines) as a map. Walking along a contour keeps the height fixed, so the slope there is zero. The direction with the largest slope is the one at right angles to the contour, and that is where the gradient points. A loss function is just such terrain over parameter space, and training walks downhill on it.
The math
- size of the nudge
- -th standard basis vector (1 in slot , 0 elsewhere)
Differentiability means is well approximated by a linear function near : , where shrinks faster than . Setting for a unit vector and dividing by gives the directional derivative:
- unit direction,
- angle between and
Everything follows from the cosine. is largest, equal to , at — so is the direction of steepest ascent and its length is the steepest slope. It is most negative at — so is steepest descent. It is zero at ; a curve inside a level set has along its tangent, so the gradient is perpendicular to level sets. At a minimum, maximum or saddle, .
Gradient descent turns this into an algorithm: repeatedly take a small step against the gradient. The first-order approximation predicts each step lowers by about .
- parameters after steps
- learning rate (step size), η > 0
- the loss being minimized
The step size has a hard limit. Along a direction where the loss has curvature (second derivative) , one step multiplies the distance to the minimum by , which shrinks only if . The steepest direction sets the limit for all of them.
See it
regime: smooth — stable iff η < 1/8 = 0.125. Click the plot to move the start.
Worked example
at . Holding fixed, ; holding fixed, . So and the steepest slope is .
In the direction the slope is , as the cosine formula requires. The direction is perpendicular to and has slope 0 — it is tangent to the level curve through .
One gradient-descent step with moves to , where , down from . The linear model predicted a drop of ; the actual drop is — the step was large enough for curvature to matter.
In code
1import numpy as np23f = lambda p: p[0]**2 + 3 * p[0] * p[1]4grad = lambda p: np.array([2 * p[0] + 3 * p[1], 3 * p[0]])56def num_grad(f, p, h=1e-5): # central difference, one coordinate at a time7 E = np.eye(len(p))8 return np.array([(f(p + h * e) - f(p - h * e)) / (2 * h) for e in E])910p = np.array([1.0, 2.0])11print(grad(p), num_grad(f, p)) # [8. 3.] [8. 3.]1213u = np.array([0.6, 0.8])14print(grad(p) @ u, np.linalg.norm(grad(p))) # 7.2 8.544... (no unit u beats ‖∇f‖)1516p_new = p - 0.1 * grad(p) # one gradient-descent step, η = 0.117print(f(p), f(p_new)) # 7.0 1.06Where it's used
Training any model
SGD, momentum and Adam all start from on a mini-batch; they differ only in how they turn it into a step.
Gradient checking
Comparing a hand-written or autograd gradient with central differences catches backprop bugs before they silently degrade training.
Saliency & attribution
shows which input features the prediction is most sensitive to.
Adversarial examples
FGSM perturbs an image by — the steepest-ascent direction under the norm — to flip a classifier's prediction.
Check yourself
At some point . Which unit direction gives ?
Check yourself
Gradient descent on converges from any start for every below which value?
Remember
- ✓ stacks the partial derivatives; .
- ✓The gradient is the steepest-ascent direction, its length the steepest slope, and it is perpendicular to level sets.
- ✓Gradient descent steps ; stability needs .
- ✓ marks minima, maxima and saddles alike; ill-conditioning makes descent zig-zag.