Derivatives & Gradients

The gradient points in the direction of steepest increase; gradient descent steps the opposite way.

6 min read606 wordscore5 viewsupdated 2026-10-03

Definition

The gradient of a differentiable function f:Rn→Rf:\mathbb{R}^n\to\mathbb{R} at a point xx is the vector of its partial derivatives, ∇f(x)=(∂f/∂x1,…,∂f/∂xn)\nabla f(x)=\big(\partial f/\partial x_1,\dots,\partial f/\partial x_n\big). Each partial derivative is the rate of change of ff when only that one coordinate moves; together they give the rate of change in every direction.

At a glance

Notation
f′(x)f'(x), dfdx\frac{df}{dx} in one variable; ∂f∂xi\frac{\partial f}{\partial x_i}, ∇f\nabla f, ∇θL\nabla_\theta L in several
Input → output
a scalar function of nn variables → a vector field in Rn\mathbb{R}^n
Key property
Duf=∇f⋅uD_uf=\nabla f\cdot u: the gradient is the steepest-ascent direction, perpendicular to level sets, and its length is the steepest slope
Prerequisites
limits and slopes, vectors & the dot product
Used in
gradient descent and every optimizer built on it, backprop, saliency maps, sensitivity analysis

The intuition

In one variable the derivative f′(x)f'(x) is the slope of the tangent line: nudge xx by a tiny hh and ff changes by about f′(x) hf'(x)\,h. With nn variables you can nudge in infinitely many directions, but differentiability means the response is linear in the nudge — so nn numbers, one slope per axis, determine all of them. Those nn numbers are the gradient.

Picture ff as terrain and its level sets (contour lines) as a map. Walking along a contour keeps the height fixed, so the slope there is zero. The direction with the largest slope is the one at right angles to the contour, and that is where the gradient points. A loss function is just such terrain over parameter space, and training walks downhill on it.

The math

f′(x)=lim⁡h→0f(x+h)−f(x)h,∂f∂xi(x)=lim⁡h→0f(x+h ei)−f(x)hf'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}{h},\qquad \frac{\partial f}{\partial x_i}(x)=\lim_{h\to0}\frac{f(x+h\,e_i)-f(x)}{h}
hh
size of the nudge
eie_i
ii-th standard basis vector (1 in slot ii, 0 elsewhere)

Differentiability means ff is well approximated by a linear function near xx: f(x+δ)=f(x)+∇f(x)⋅δ+o(∥δ∥)f(x+\delta)=f(x)+\nabla f(x)\cdot\delta+o(\lVert\delta\rVert), where o(∥δ∥)o(\lVert\delta\rVert) shrinks faster than ∥δ∥\lVert\delta\rVert. Setting δ=h u\delta=h\,u for a unit vector uu and dividing by hh gives the directional derivative:

Duf(x)=lim⁡h→0f(x+hu)−f(x)h=∇f(x)⋅u=∥∇f(x)∥cos⁡θD_uf(x)=\lim_{h\to0}\frac{f(x+hu)-f(x)}{h}=\nabla f(x)\cdot u=\lVert\nabla f(x)\rVert\cos\theta
uu
unit direction, ∥u∥=1\lVert u\rVert=1
θ\theta
angle between uu and ∇f(x)\nabla f(x)

Everything follows from the cosine. DufD_uf is largest, equal to ∥∇f∥\lVert\nabla f\rVert, at θ=0\theta=0 — so ∇f\nabla f is the direction of steepest ascent and its length is the steepest slope. It is most negative at θ=180°\theta=180° — so −∇f-\nabla f is steepest descent. It is zero at θ=90°\theta=90°; a curve inside a level set has Duf=0D_uf=0 along its tangent, so the gradient is perpendicular to level sets. At a minimum, maximum or saddle, ∇f=0\nabla f=0.

Gradient descent turns this into an algorithm: repeatedly take a small step against the gradient. The first-order approximation predicts each step lowers ff by about η∥∇f∥2\eta\lVert\nabla f\rVert^2.

θt+1=θt−η ∇L(θt)\theta_{t+1}=\theta_t-\eta\,\nabla L(\theta_t)
θt\theta_t
parameters after tt steps
η\eta
learning rate (step size), η > 0
LL
the loss being minimized

The step size has a hard limit. Along a direction where the loss has curvature (second derivative) λ\lambda, one step multiplies the distance to the minimum by (1−ηλ)(1-\eta\lambda), which shrinks only if 0<η<2/λ0<\eta<2/\lambda. The steepest direction sets the limit for all of them.

See it

∇fudrag the point · drag u
point = (-1.90, 0.35)f = 0.492‖∇f‖ = 0.687∂f/∂x = -0.610∂f/∂y = 0.315∠(u, ∇f) = 93°
Duf = ∇f·u = -0.030
fig.A tilted bowl plus a Gaussian bump, on equal axes. At the starting point (−1.90, 0.35): ∂f/∂x = −0.610, ∂f/∂y = 0.315, ‖∇f‖ = 0.687, and the orange ∇f meets the green contour at a right angle. u starts 93° from ∇f, so D_u f = −0.030 — nearly zero, because u almost follows the contour. Press u ∥ ∇f and D_u f becomes exactly ‖∇f‖. The small arrows vanish at the minimum near (−0.64, −0.03).
f(x, y) = x² + 8y²step 00 · loss 58.9700

regime: smooth — stable iff η < 1/8 = 0.125. Click the plot to move the start.

fig.Gradient descent on f(x, y) = x² + 8y² from (−4.5, 2.2). The curvature along y is 16, so η past 1/16 makes y oscillate and η past 1/8 = 2/16 diverges. Click the plot to move the start.

Worked example

f(x,y)=x2+3xyf(x,y)=x^2+3xy at p=(1,2)p=(1,2). Holding yy fixed, ∂f/∂x=2x+3y=8\partial f/\partial x=2x+3y=8; holding xx fixed, ∂f/∂y=3x=3\partial f/\partial y=3x=3. So ∇f(p)=(8,3)\nabla f(p)=(8,3) and the steepest slope is ∥∇f∥=73≈8.544\lVert\nabla f\rVert=\sqrt{73}\approx8.544.

In the direction u=(0.6,0.8)u=(0.6,0.8) the slope is ∇f⋅u=4.8+2.4=7.2<8.544\nabla f\cdot u=4.8+2.4=7.2<8.544, as the cosine formula requires. The direction u=(3,−8)/73u=(3,-8)/\sqrt{73} is perpendicular to ∇f\nabla f and has slope 0 — it is tangent to the level curve f=7f=7 through pp.

One gradient-descent step with η=0.1\eta=0.1 moves to p−0.1 (8,3)=(0.2,1.7)p-0.1\,(8,3)=(0.2,1.7), where f=0.04+1.02=1.06f=0.04+1.02=1.06, down from f(p)=7f(p)=7. The linear model predicted a drop of η∥∇f∥2=7.3\eta\lVert\nabla f\rVert^2=7.3; the actual drop is 5.945.94 — the step was large enough for curvature to matter.

In code

python
1import numpy as np
2
3f = lambda p: p[0]**2 + 3 * p[0] * p[1]
4grad = lambda p: np.array([2 * p[0] + 3 * p[1], 3 * p[0]])
5
6def num_grad(f, p, h=1e-5): # central difference, one coordinate at a time
7 E = np.eye(len(p))
8 return np.array([(f(p + h * e) - f(p - h * e)) / (2 * h) for e in E])
9
10p = np.array([1.0, 2.0])
11print(grad(p), num_grad(f, p)) # [8. 3.] [8. 3.]
12
13u = np.array([0.6, 0.8])
14print(grad(p) @ u, np.linalg.norm(grad(p))) # 7.2 8.544... (no unit u beats ‖∇f‖)
15
16p_new = p - 0.1 * grad(p) # one gradient-descent step, η = 0.1
17print(f(p), f(p_new)) # 7.0 1.06
Analytic gradient checked against central differences, the directional-derivative bound, and one descent step.

Where it's used

Training any model

SGD, momentum and Adam all start from ∇θL\nabla_\theta L on a mini-batch; they differ only in how they turn it into a step.

Gradient checking

Comparing a hand-written or autograd gradient with central differences catches backprop bugs before they silently degrade training.

Saliency & attribution

∂(class score)/∂(pixel)\partial(\text{class score})/\partial(\text{pixel}) shows which input features the prediction is most sensitive to.

Adversarial examples

FGSM perturbs an image by ε sign(∇xL)\varepsilon\,\mathrm{sign}(\nabla_x L) — the steepest-ascent direction under the L∞L_\infty norm — to flip a classifier's prediction.

Check yourself

At some point ∇f=(3,4)\nabla f=(3,4). Which unit direction uu gives Duf=0D_uf=0?

Check yourself

Gradient descent on f(x)=5x2f(x)=5x^2 converges from any start for every η\eta below which value?

Remember

  • ✓∇f\nabla f stacks the partial derivatives; Duf=∇f⋅u=∥∇f∥cos⁡θD_uf=\nabla f\cdot u=\lVert\nabla f\rVert\cos\theta.
  • ✓The gradient is the steepest-ascent direction, its length the steepest slope, and it is perpendicular to level sets.
  • ✓Gradient descent steps θ←θ−η∇L\theta\leftarrow\theta-\eta\nabla L; stability needs η<2/λmax⁡\eta<2/\lambda_{\max}.
  • ✓∇f=0\nabla f=0 marks minima, maxima and saddles alike; ill-conditioning makes descent zig-zag.
← back to Mathematics roadmap