Jacobian & Hessian

The Jacobian collects all first derivatives of a vector function; the Hessian collects second derivatives and encodes curvature.

6 min read534 wordsadvanced5 viewsupdated 2026-10-03

Definition

Jacobian and Hessian For F:Rn→RmF:\mathbb{R}^n\to\mathbb{R}^m, the Jacobian JF(x)J_F(x) is the m×nm\times n matrix of partial derivatives ∂Fi/∂xj\partial F_i/\partial x_j: the best linear approximation of FF near xx. For a scalar f:Rn→Rf:\mathbb{R}^n\to\mathbb{R}, the Hessian ∇2f(x)\nabla^2 f(x) is the n×nn\times n matrix of second partials ∂2f/∂xi∂xj\partial^2 f/\partial x_i\partial x_j — the Jacobian of the gradient — and its eigenvalues are the curvatures of ff along its principal directions.

At a glance

Notation
JF(x)J_F(x) or ∂F/∂x\partial F/\partial x (m×nm\times n); ∇2f(x)\nabla^2 f(x) or HH (n×nn\times n, symmetric)
Input → output
JJ: vector function → matrix of first derivatives; HH: scalar function → matrix of second derivatives
Key property
F(x+δ)≈F(x)+JδF(x+\delta)\approx F(x)+J\delta; ∣det⁡J∣|\det J| is the local volume scale; the signs of HH's eigenvalues classify a critical point
Cost
a full HH needs n2n^2 entries; a Hessian-vector product HvHv costs about two gradients
Prerequisites
derivatives & gradients, chain rule, eigenvectors
Used in
backprop, normalizing flows, Newton's method, saddle-point analysis, Laplace approximation

The intuition

Zoom in far enough on a smooth map and it looks linear: a tiny square becomes a tiny parallelogram. The Jacobian is the matrix of that linear map. Its columns say where the two edges of the square go, and its determinant says how much the area grew or shrank.

The gradient says which way is downhill; the Hessian says how the slope itself changes — whether the ground curves up like a bowl, down like a dome, or up one way and down another like a saddle. Each eigenvector of HH is a direction, and its eigenvalue is the curvature along it.

The math

JF(x)=[∂F1∂x1⋯∂F1∂xn⋮⋱⋮∂Fm∂x1⋯∂Fm∂xn],F(x+δ)=F(x)+JF(x) δ+o(∥δ∥)J_F(x)=\begin{bmatrix}\dfrac{\partial F_1}{\partial x_1}&\cdots&\dfrac{\partial F_1}{\partial x_n}\\ \vdots&\ddots&\vdots\\ \dfrac{\partial F_m}{\partial x_1}&\cdots&\dfrac{\partial F_m}{\partial x_n}\end{bmatrix},\qquad F(x+\delta)=F(x)+J_F(x)\,\delta+o(\|\delta\|)
FiF_i
ii-th output component of FF; row ii of JJ is ∇Fi⊤\nabla F_i^{\top}
δ\delta
small displacement of the input xx
o(∥δ∥)o(\|\delta\|)
an error that vanishes faster than ∥δ∥\|\delta\|

Chain rule. Jacobians compose by matrix multiplication: JG∘F(x)=JG(F(x)) JF(x)J_{G\circ F}(x)=J_G(F(x))\,J_F(x). Backpropagation evaluates this product right-to-left as vector–Jacobian products v⊤Jv^\top J, never forming the matrices. Change of variables. If y=F(x)y=F(x) is invertible, densities transform as pY(y)=pX(x)/∣det⁡JF(x)∣p_Y(y)=p_X(x)/|\det J_F(x)| — the volume factor normalizing flows are built on.

Hij=∂2f∂xi ∂xj,f(x+δ)≈f(x)+∇f(x)⊤δ+12 δ⊤H δH_{ij}=\frac{\partial^2 f}{\partial x_i\,\partial x_j},\qquad f(x+\delta)\approx f(x)+\nabla f(x)^{\top}\delta+\tfrac12\,\delta^{\top}H\,\delta
HH
Hessian at xx; symmetric (Hij=HjiH_{ij}=H_{ji}) when ff has continuous second derivatives
∇f(x)\nabla f(x)
gradient — the Jacobian of a scalar function, transposed

Second-derivative test. At a critical point (∇f=0\nabla f=0) the quadratic term decides. Write H=QΛQ⊤H=Q\Lambda Q^\top with eigenvalues λi\lambda_i: all λi>0\lambda_i>0 → strict local minimum; all <0<0 → maximum; mixed signs → saddle; some λi=0\lambda_i=0 → degenerate, the test is inconclusive.

Curvature and optimization. On f=12x⊤Hxf=\tfrac12x^\top Hx, gradient descent multiplies the component along eigenvector qiq_i by (1−ηλi)(1-\eta\lambda_i) per step. It converges only if η<2/λmax⁡\eta<2/\lambda_{\max}, and progress along the flattest direction is limited by the condition number κ=λmax⁡/λmin⁡\kappa=\lambda_{\max}/\lambda_{\min}. Newton's method x←x−H−1∇fx\leftarrow x-H^{-1}\nabla f rescales every direction by its own curvature and solves a quadratic in one step.

See it

input (x, y) · shade = det Joutput F(x, y) — the warped grid
J =1.000.530.421.00

at (x, y) = (0.80, -0.50) · det J = 0.780

true area ratio of the curved image = 0.794 · gap 0.014

F(x, y) = (x + 0.6 sin y, y + 0.6 sin x) · drag the white point · green = true image · dashed = J × square (linear approximation)

fig.F(x, y) = (x + 0.6 sin y, y + 0.6 sin x). At (0.8, −0.5), J = [[1, 0.53], [0.42, 1]] and det J = 0.780; the true image of a 0.9-wide square covers 0.794 of its area. Shrink the square to 0.2 and the curved image and the dashed parallelogram J·square coincide (gap 0.001). The left shading is det J: orange where area grows, blue where it shrinks.
λ₁ = 1.5λ₂ = -0.6

Hessian

1.120.800.80-0.22

det = λ₁λ₂ = -0.90 · tr = 0.90

saddle — H indefinite

GD (η = 0.25) from the white dot: escapes along the λ₂ direction

f(x) = ½ xᵀHx · orange contours f > 0, blue f < 0

fig.f = ½ xᵀHx with H built from its eigenvalues and a rotation. The default λ₁ = 1.5, λ₂ = −0.6 at 25° gives H = [[1.12, 0.80], [0.80, −0.22]], det = −0.90: a saddle. Gradient descent (η = 0.25) shrinks the λ₁ component by 0.625 per step but grows the λ₂ component by 1.15 per step, so it escapes along the negative-curvature direction. Make both eigenvalues positive for a bowl, both negative for a dome, one zero for a degenerate trough.

Worked example

Jacobian. F(x,y)=(x2y, x+3y)F(x,y)=(x^2y,\ x+3y) has J=[2xyx213]J=\begin{bmatrix}2xy&x^2\\1&3\end{bmatrix}. At (1,2)(1,2): J=[4113]J=\begin{bmatrix}4&1\\1&3\end{bmatrix}, det⁡J=11\det J=11 — a tiny square around (1,2)(1,2) maps to a parallelogram 11 times its area. Polar coordinates (r,φ)↦(rcos⁡φ,rsin⁡φ)(r,\varphi)\mapsto(r\cos\varphi,r\sin\varphi) have det⁡J=r\det J=r, the rr in dx dy=r dr dφdx\,dy=r\,dr\,d\varphi.

Hessian. f(x,y)=x3−3xy+y3f(x,y)=x^3-3xy+y^3 has ∇f=(3x2−3y, 3y2−3x)\nabla f=(3x^2-3y,\ 3y^2-3x), zero at (0,0)(0,0) and (1,1)(1,1), and H=[6x−3−36y]H=\begin{bmatrix}6x&-3\\-3&6y\end{bmatrix}. At (0,0)(0,0): H=[0−3−30]H=\begin{bmatrix}0&-3\\-3&0\end{bmatrix} with eigenvalues ±3\pm3 — a saddle. At (1,1)(1,1): H=[6−3−36]H=\begin{bmatrix}6&-3\\-3&6\end{bmatrix} with eigenvalues 33 and 99 — a local minimum, f=−1f=-1, with condition number 9/3=39/3=3.

In code

python
1import torch
2from torch.autograd.functional import jacobian, hessian, hvp
3
4F = lambda v: torch.stack([v[0]**2 * v[1], v[0] + 3 * v[1]])
5J = jacobian(F, torch.tensor([1.0, 2.0])) # tensor([[4., 1.], [1., 3.]])
6torch.linalg.det(J) # 11.0
7
8f = lambda v: v[0]**3 - 3 * v[0] * v[1] + v[1]**3
9H = hessian(f, torch.tensor([1.0, 1.0])) # tensor([[ 6., -3.], [-3., 6.]])
10torch.linalg.eigvalsh(H) # tensor([3., 9.]) -> minimum
11
12_, Hv = hvp(f, torch.tensor([1.0, 1.0]), torch.tensor([1.0, 0.0]))
13Hv # tensor([ 6., -3.]) = first column of H
torch.autograd.functional builds full Jacobians and Hessians for small problems; hvp gives Hv without materializing H, which is what large-scale second-order methods use.

Where it's used

Backpropagation

Every layer is a Jacobian in the chain JL⋯J1J_L\cdots J_1; reverse-mode autodiff computes v⊤Jv^\top J products layer by layer. The softmax Jacobian diag(s)−ss⊤\mathrm{diag}(s)-ss^\top is why cross-entropy's gradient collapses to s−ys-y.

Normalizing flows

Flows stack invertible maps and track log⁡∣det⁡J∣\log|\det J| to evaluate exact likelihoods; architectures (coupling layers) are designed so JJ is triangular and the determinant is a product of diagonal entries.

Second-order optimization

Newton and quasi-Newton methods (L-BFGS, the default in scikit-learn's logistic regression) use or approximate H−1H^{-1}; logistic regression's Hessian is X⊤SXX^\top SX with S=diag(pi(1−pi))S=\mathrm{diag}(p_i(1-p_i)).

Loss-landscape and uncertainty

The top Hessian eigenvalue measures sharpness and caps the stable learning rate (η<2/λmax⁡\eta<2/\lambda_{\max}); the Laplace approximation uses H−1H^{-1} at the optimum as a posterior covariance.

Check yourself

F:R3→R2F:\mathbb{R}^3\to\mathbb{R}^2. The shape of its Jacobian is…

Check yourself

At a point with ∇f=0\nabla f=0, the Hessian has eigenvalues 44 and −1-1. The point is…

Remember

  • ✓JFJ_F (m×nm\times n) is the local linear map: F(x+δ)≈F(x)+JδF(x+\delta)\approx F(x)+J\delta; ∣det⁡J∣|\det J| scales volume.
  • ✓Jacobians multiply under composition — the chain rule and backprop.
  • ✓HH is symmetric; at a critical point its eigenvalue signs classify min, max, saddle or degenerate.
  • ✓Curvature sets the step size (η<2/λmax⁡\eta<2/\lambda_{\max}) and speed (κ\kappa); use HvHv products, not HH, at scale.
← back to Mathematics roadmap