Definition
Jacobian and Hessian For , the Jacobian is the matrix of partial derivatives : the best linear approximation of near . For a scalar , the Hessian is the matrix of second partials — the Jacobian of the gradient — and its eigenvalues are the curvatures of along its principal directions.
At a glance
- Notation
- or (); or (, symmetric)
- Input → output
- : vector function → matrix of first derivatives; : scalar function → matrix of second derivatives
- Key property
- ; is the local volume scale; the signs of 's eigenvalues classify a critical point
- Cost
- a full needs entries; a Hessian-vector product costs about two gradients
- Prerequisites
- derivatives & gradients, chain rule, eigenvectors
- Used in
- backprop, normalizing flows, Newton's method, saddle-point analysis, Laplace approximation
The intuition
Zoom in far enough on a smooth map and it looks linear: a tiny square becomes a tiny parallelogram. The Jacobian is the matrix of that linear map. Its columns say where the two edges of the square go, and its determinant says how much the area grew or shrank.
The gradient says which way is downhill; the Hessian says how the slope itself changes — whether the ground curves up like a bowl, down like a dome, or up one way and down another like a saddle. Each eigenvector of is a direction, and its eigenvalue is the curvature along it.
The math
- -th output component of ; row of is
- small displacement of the input
- an error that vanishes faster than
Chain rule. Jacobians compose by matrix multiplication: . Backpropagation evaluates this product right-to-left as vector–Jacobian products , never forming the matrices. Change of variables. If is invertible, densities transform as — the volume factor normalizing flows are built on.
- Hessian at ; symmetric () when has continuous second derivatives
- gradient — the Jacobian of a scalar function, transposed
Second-derivative test. At a critical point () the quadratic term decides. Write with eigenvalues : all → strict local minimum; all → maximum; mixed signs → saddle; some → degenerate, the test is inconclusive.
Curvature and optimization. On , gradient descent multiplies the component along eigenvector by per step. It converges only if , and progress along the flattest direction is limited by the condition number . Newton's method rescales every direction by its own curvature and solves a quadratic in one step.
See it
at (x, y) = (0.80, -0.50) · det J = 0.780
true area ratio of the curved image = 0.794 · gap 0.014
F(x, y) = (x + 0.6 sin y, y + 0.6 sin x) · drag the white point · green = true image · dashed = J × square (linear approximation)
Hessian
det = λ₁λ₂ = -0.90 · tr = 0.90
saddle — H indefinite
GD (η = 0.25) from the white dot: escapes along the λ₂ direction
f(x) = ½ xᵀHx · orange contours f > 0, blue f < 0
Worked example
Jacobian. has . At : , — a tiny square around maps to a parallelogram 11 times its area. Polar coordinates have , the in .
Hessian. has , zero at and , and . At : with eigenvalues — a saddle. At : with eigenvalues and — a local minimum, , with condition number .
In code
1import torch2from torch.autograd.functional import jacobian, hessian, hvp34F = lambda v: torch.stack([v[0]**2 * v[1], v[0] + 3 * v[1]])5J = jacobian(F, torch.tensor([1.0, 2.0])) # tensor([[4., 1.], [1., 3.]])6torch.linalg.det(J) # 11.078f = lambda v: v[0]**3 - 3 * v[0] * v[1] + v[1]**39H = hessian(f, torch.tensor([1.0, 1.0])) # tensor([[ 6., -3.], [-3., 6.]])10torch.linalg.eigvalsh(H) # tensor([3., 9.]) -> minimum1112_, Hv = hvp(f, torch.tensor([1.0, 1.0]), torch.tensor([1.0, 0.0]))13Hv # tensor([ 6., -3.]) = first column of HWhere it's used
Backpropagation
Every layer is a Jacobian in the chain ; reverse-mode autodiff computes products layer by layer. The softmax Jacobian is why cross-entropy's gradient collapses to .
Normalizing flows
Flows stack invertible maps and track to evaluate exact likelihoods; architectures (coupling layers) are designed so is triangular and the determinant is a product of diagonal entries.
Second-order optimization
Newton and quasi-Newton methods (L-BFGS, the default in scikit-learn's logistic regression) use or approximate ; logistic regression's Hessian is with .
Loss-landscape and uncertainty
The top Hessian eigenvalue measures sharpness and caps the stable learning rate (); the Laplace approximation uses at the optimum as a posterior covariance.
Check yourself
. The shape of its Jacobian is…
Check yourself
At a point with , the Hessian has eigenvalues and . The point is…
Remember
- ✓ () is the local linear map: ; scales volume.
- ✓Jacobians multiply under composition — the chain rule and backprop.
- ✓ is symmetric; at a critical point its eigenvalue signs classify min, max, saddle or degenerate.
- ✓Curvature sets the step size () and speed (); use products, not , at scale.