Definition
Gradient descent is the iteration : from the current point, take a step of size (the learning rate) in the direction in which decreases fastest. Repeat until the gradient is (nearly) zero.
Every optimizer used to train neural networks is a modification of this one rule. SGD and mini-batch training replace the exact gradient by a cheap estimate, momentum smooths it, Adam rescales it per parameter. Their behaviour makes sense only once you understand how plain gradient descent interacts with the curvature of the loss, and that is the subject of this article.
The method is old. Augustin-Louis Cauchy described it in 1847, in a short note on solving systems of simultaneous equations by repeatedly moving downhill on a sum of squared residuals. Its convergence theory for smooth convex functions was worked out in the 1960s, notably by Polyak, and Nesterov later proved that for this function class no first-order method can beat a rate of order where gradient descent pays , which is why accelerated methods exist (see Momentum and Nesterov accelerated gradient).
The update rule and why it points downhill
- all trainable parameters, as one vector
- the gradient: the vector of partial derivatives , which points in the direction of steepest ascent
- learning rate (step size),
- a penalty for moving far from where the linear approximation was taken; small = little trust in it
The right-hand form shows what a step really is: minimize the first-order Taylor model of around , but stay close because the model is only locally accurate. Setting its derivative to zero gives exactly the left-hand rule. Among all directions of unit length, makes the directional derivative most negative, so this is steepest descent in the Euclidean norm. Change the norm and you change the method: preconditioned and adaptive methods are steepest descent in a different geometry.
Curvature caps the step: L-smoothness
The linear model ignores curvature, and curvature is what makes a step overshoot. A function is -smooth if its gradient cannot change faster than times the distance moved; for a twice-differentiable this means every eigenvalue of the Hessian is at most in absolute value. Smoothness gives a quadratic upper bound on , and plugging the gradient step into it gives the descent lemma:
- smoothness constant = largest curvature, of the Hessian
- positive exactly when : then every step is guaranteed to decrease
- maximizes the guaranteed decrease, which is then
So is the "safe" step and the edge. The decrease is proportional to : gradient descent moves fast on steep slopes and slows to a crawl where the gradient is small, which is exactly where it spends most of its time on ill-conditioned problems.
Exact behaviour on a quadratic
Near a minimum, any smooth loss looks like a quadratic , so the quadratic tells the whole local story. Its gradient is , and writing the error in the eigenbasis of decouples the iteration into one scalar recursion per eigen-direction:
- eigenvalues of the Hessian : the curvature along each principal direction; ,
- per-step multiplier of that direction: in monotone, in zigzag, below divergent
Every direction shrinks at its own rate, and the slowest one sets the pace: . Raising helps the flat direction but hurts the steep one. The two balance at , which gives the best possible constant-step rate:
- condition number: how much steeper the steepest direction is than the flattest
- steps needed: linear in , logarithmic in the accuracy
Gradient descent at a glance
- Update
- , one gradient per step
- Stability
- (on a quadratic: )
- Safe default
- : monotone decrease, guaranteed drop per step
- Convex, L-smooth
- with : rate
- μ-strongly convex
- with : linear (geometric) rate
- Best constant step
- , error contracts by per step
- Non-convex
- with , : finds a stationary point, not necessarily a minimum
Worked example: the figure's first steps
Take the figure's defaults: , (), , start , so . The gradient is .
| t | θₜ = (x, y) | ∇f(θₜ) | f(θₜ) |
|---|---|---|---|
| 0 | (−4.000, 1.500) | (−4.00, 15.00) | 19.250 |
| 1 | (−3.400, −0.750) | (−3.40, −7.50) | 8.593 |
| 2 | (−2.890, 0.375) | (−2.89, 3.75) | 4.879 |
| 3 | (−2.457, −0.188) | (−2.46, −1.88) | 3.193 |
- 1The -component of the gradient is 15, almost 4 times the -component, so the first step is mostly across the valley: goes from to . It overshot the floor, because .
- 2Each step multiplies by and by . After 10 steps is , while is still .
- 3From then on the flat direction is everything: the loss falls by per step, and reaching of the start takes 40 steps in total.
- 4Change only to : the multiplier becomes , its magnitude grows 10% per step, and the loss passes at step 60 although the direction was converging the whole time.
What the condition number means in practice
For a linear model with squared loss the Hessian is the feature covariance , so is a property of the data. Features on very different scales, or strongly correlated features, give a large , and standardizing them (zero mean, unit variance) is the cheapest way to make gradient descent faster. Normalization layers, careful initialization and residual connections play a similar role in deep networks; adaptive optimizers and second-order methods attack directly.
Neural-network losses are not quadratic, and the Hessian changes along the path. Cohen et al. (2021) found that full-batch gradient descent on neural networks typically drives the largest Hessian eigenvalue (the sharpness) up until it hovers around , the edge of stability, and training keeps making progress there, non-monotonically. The rule is still the right mental model; the network just adapts its own to the step size.
1import torch23# f(θ) = ½(μ x² + L y²) with μ = 1, L = 104H = torch.tensor([1.0, 10.0])5f = lambda th: 0.5 * (H * th**2).sum()67theta = torch.tensor([-4.0, 1.5], requires_grad=True)8eta = 0.159for t in range(3):10 loss = f(theta)11 (g,) = torch.autograd.grad(loss, theta)12 with torch.no_grad():13 theta -= eta * g # θ ← θ − η ∇f14 print(t + 1, theta.tolist(), f(theta).item())15# ≈ 1 [-3.4, -0.75] 8.5925 2 [-2.89, 0.375] 4.8792 3 [-2.4565, -0.1875] 3.1931617# The same update through the optimizer API (full gradient = gradient descent)18theta = torch.tensor([-4.0, 1.5], requires_grad=True)19opt = torch.optim.SGD([theta], lr=0.15) # no momentum: plain GD step20for _ in range(3):21 opt.zero_grad()22 f(theta).backward()23 opt.step()2425# Estimate L = λ_max(Hessian) without forming the Hessian: power iteration on H·v26def top_eigenvalue(loss_fn, params, iters=20):27 v = [torch.randn_like(p) for p in params]28 for _ in range(iters):29 norm = torch.sqrt(sum((u**2).sum() for u in v))30 v = [u / norm for u in v]31 grads = torch.autograd.grad(loss_fn(), params, create_graph=True)32 Hv = torch.autograd.grad(grads, params, grad_outputs=v)33 lam = sum((h * u).sum() for h, u in zip(Hv, v)).item() # Rayleigh quotient34 v = [h.detach() for h in Hv]35 return lam3637print(top_eigenvalue(lambda: f(theta), [theta])) # ≈ 10.0, so keep η below 2/10 = 0.2Use when
- +One hyperparameter, one gradient per step, no extra state: the simplest possible optimizer
- +Guaranteed monotone decrease for , with clean convergence rates on convex problems
- +Works for any differentiable loss; the template every other optimizer modifies
Avoid when
- −Steps needed grow linearly with the condition number ; slow on ill-conditioned problems
- −The best depends on curvature you usually do not know; too large diverges
- −Exact gradients of a large dataset are expensive (see batch vs stochastic variants)
- −On non-convex losses it finds a stationary point, which can be a saddle or a poor local minimum
Check yourself
The largest Hessian eigenvalue of your loss near the minimum is 50. Which learning rate is the largest one for which gradient descent still converges there?
Check yourself
On f = ½(x² + 100y²) you use the best constant step η = 2/(L+μ). Roughly how much does the error shrink per step?
Remember
- ✓Update: , the minimizer of the linear model plus a proximity penalty .
- ✓-smoothness bounds curvature; every step decreases when , and is the safe default.
- ✓On a quadratic each eigen-direction is multiplied by : divergence past , slow progress along .
- ✓Best constant rate ; steps grow like .
- ✓Convex: ; strongly convex: linear ; non-convex: gradient norm .