# Jacobian & Hessian

> The Jacobian collects all first derivatives of a vector function; the Hessian collects second derivatives and encodes curvature.

Source: https://www.mlfoundry.in/learn/jacobian-hessian  
Section: Mathematics › Calculus & Optimization  
Level: advanced · 6 min read · updated 2026-10-03

**Jacobian and Hessian:** For $F:\mathbb{R}^n\to\mathbb{R}^m$, the **Jacobian** $J_F(x)$ is the $m\times n$ matrix of partial derivatives $\partial F_i/\partial x_j$: the best linear approximation of $F$ near $x$. For a scalar $f:\mathbb{R}^n\to\mathbb{R}$, the **Hessian** $\nabla^2 f(x)$ is the $n\times n$ matrix of second partials $\partial^2 f/\partial x_i\partial x_j$ — the Jacobian of the gradient — and its eigenvalues are the curvatures of $f$ along its principal directions.

- **Notation:** $J_F(x)$ or $\partial F/\partial x$ ($m\times n$); $\nabla^2 f(x)$ or $H$ ($n\times n$, symmetric)
- **Input → output:** $J$: vector function → matrix of first derivatives; $H$: scalar function → matrix of second derivatives
- **Key property:** $F(x+\delta)\approx F(x)+J\delta$; $|\det J|$ is the local volume scale; the signs of $H$'s eigenvalues classify a critical point
- **Cost:** a full $H$ needs $n^2$ entries; a Hessian-vector product $Hv$ costs about two gradients
- **Prerequisites:** derivatives & gradients, chain rule, eigenvectors
- **Used in:** backprop, normalizing flows, Newton's method, saddle-point analysis, Laplace approximation

## The intuition

Zoom in far enough on a smooth map and it looks linear: a tiny square becomes a tiny parallelogram. The Jacobian is the matrix of that linear map. Its columns say where the two edges of the square go, and its determinant says how much the area grew or shrank.

The gradient says which way is downhill; the Hessian says how the slope itself changes — whether the ground curves up like a bowl, down like a dome, or up one way and down another like a saddle. Each eigenvector of $H$ is a direction, and its eigenvalue is the curvature along it.

## The math

$$
J_F(x)=\begin{bmatrix}\dfrac{\partial F_1}{\partial x_1}&\cdots&\dfrac{\partial F_1}{\partial x_n}\\ \vdots&\ddots&\vdots\\ \dfrac{\partial F_m}{\partial x_1}&\cdots&\dfrac{\partial F_m}{\partial x_n}\end{bmatrix},\qquad F(x+\delta)=F(x)+J_F(x)\,\delta+o(\|\delta\|)
$$

- $F_i$: $i$-th output component of $F$; row $i$ of $J$ is $\nabla F_i^{\top}$
- $\delta$: small displacement of the input $x$
- $o(\|\delta\|)$: an error that vanishes faster than $\|\delta\|$

**Chain rule.** Jacobians compose by matrix multiplication: $J_{G\circ F}(x)=J_G(F(x))\,J_F(x)$. Backpropagation evaluates this product right-to-left as vector–Jacobian products $v^\top J$, never forming the matrices. **Change of variables.** If $y=F(x)$ is invertible, densities transform as $p_Y(y)=p_X(x)/|\det J_F(x)|$ — the volume factor normalizing flows are built on.

$$
H_{ij}=\frac{\partial^2 f}{\partial x_i\,\partial x_j},\qquad f(x+\delta)\approx f(x)+\nabla f(x)^{\top}\delta+\tfrac12\,\delta^{\top}H\,\delta
$$

- $H$: Hessian at $x$; symmetric ($H_{ij}=H_{ji}$) when $f$ has continuous second derivatives
- $\nabla f(x)$: gradient — the Jacobian of a scalar function, transposed

**Second-derivative test.** At a critical point ($\nabla f=0$) the quadratic term decides. Write $H=Q\Lambda Q^\top$ with eigenvalues $\lambda_i$: all $\lambda_i>0$ → strict local **minimum**; all $<0$ → **maximum**; mixed signs → **saddle**; some $\lambda_i=0$ → degenerate, the test is inconclusive.

**Curvature and optimization.** On $f=\tfrac12x^\top Hx$, gradient descent multiplies the component along eigenvector $q_i$ by $(1-\eta\lambda_i)$ per step. It converges only if $\eta<2/\lambda_{\max}$, and progress along the flattest direction is limited by the condition number $\kappa=\lambda_{\max}/\lambda_{\min}$. Newton's method $x\leftarrow x-H^{-1}\nabla f$ rescales every direction by its own curvature and solves a quadratic in one step.

## See it

> Figure: F(x, y) = (x + 0.6 sin y, y + 0.6 sin x). At (0.8, −0.5), J = [[1, 0.53], [0.42, 1]] and det J = 0.780; the true image of a 0.9-wide square covers 0.794 of its area. Shrink the square to 0.2 and the curved image and the dashed parallelogram J·square coincide (gap 0.001). The left shading is det J: orange where area grows, blue where it shrinks.

> Figure: f = ½ xᵀHx with H built from its eigenvalues and a rotation. The default λ₁ = 1.5, λ₂ = −0.6 at 25° gives H = [[1.12, 0.80], [0.80, −0.22]], det = −0.90: a saddle. Gradient descent (η = 0.25) shrinks the λ₁ component by 0.625 per step but grows the λ₂ component by 1.15 per step, so it escapes along the negative-curvature direction. Make both eigenvalues positive for a bowl, both negative for a dome, one zero for a degenerate trough.

## Worked example

**Jacobian.** $F(x,y)=(x^2y,\ x+3y)$ has $J=\begin{bmatrix}2xy&x^2\\1&3\end{bmatrix}$. At $(1,2)$: $J=\begin{bmatrix}4&1\\1&3\end{bmatrix}$, $\det J=11$ — a tiny square around $(1,2)$ maps to a parallelogram 11 times its area. Polar coordinates $(r,\varphi)\mapsto(r\cos\varphi,r\sin\varphi)$ have $\det J=r$, the $r$ in $dx\,dy=r\,dr\,d\varphi$.

**Hessian.** $f(x,y)=x^3-3xy+y^3$ has $\nabla f=(3x^2-3y,\ 3y^2-3x)$, zero at $(0,0)$ and $(1,1)$, and $H=\begin{bmatrix}6x&-3\\-3&6y\end{bmatrix}$. At $(0,0)$: $H=\begin{bmatrix}0&-3\\-3&0\end{bmatrix}$ with eigenvalues $\pm3$ — a saddle. At $(1,1)$: $H=\begin{bmatrix}6&-3\\-3&6\end{bmatrix}$ with eigenvalues $3$ and $9$ — a local minimum, $f=-1$, with condition number $9/3=3$.

## In code

```python
import torch
from torch.autograd.functional import jacobian, hessian, hvp

F = lambda v: torch.stack([v[0]**2 * v[1], v[0] + 3 * v[1]])
J = jacobian(F, torch.tensor([1.0, 2.0]))     # tensor([[4., 1.], [1., 3.]])
torch.linalg.det(J)                            # 11.0

f = lambda v: v[0]**3 - 3 * v[0] * v[1] + v[1]**3
H = hessian(f, torch.tensor([1.0, 1.0]))      # tensor([[ 6., -3.], [-3.,  6.]])
torch.linalg.eigvalsh(H)                       # tensor([3., 9.])  -> minimum

_, Hv = hvp(f, torch.tensor([1.0, 1.0]), torch.tensor([1.0, 0.0]))
Hv                                             # tensor([ 6., -3.])  = first column of H
```

torch.autograd.functional builds full Jacobians and Hessians for small problems; hvp gives Hv without materializing H, which is what large-scale second-order methods use.

- **Backpropagation:** Every layer is a Jacobian in the chain $J_L\cdots J_1$; reverse-mode autodiff computes $v^\top J$ products layer by layer. The softmax Jacobian $\mathrm{diag}(s)-ss^\top$ is why cross-entropy's gradient collapses to $s-y$.
- **Normalizing flows:** Flows stack invertible maps and track $\log|\det J|$ to evaluate exact likelihoods; architectures (coupling layers) are designed so $J$ is triangular and the determinant is a product of diagonal entries.
- **Second-order optimization:** Newton and quasi-Newton methods (L-BFGS, the default in scikit-learn's logistic regression) use or approximate $H^{-1}$; logistic regression's Hessian is $X^\top SX$ with $S=\mathrm{diag}(p_i(1-p_i))$.
- **Loss-landscape and uncertainty:** The top Hessian eigenvalue measures sharpness and caps the stable learning rate ($\eta<2/\lambda_{\max}$); the Laplace approximation uses $H^{-1}$ at the optimum as a posterior covariance.

> **Never form the Hessian of a large model:** A model with $10^8$ parameters has a Hessian with $10^{16}$ entries. Work with Hessian-vector products (two backward passes) and iterative methods — power iteration for $\lambda_{\max}$, conjugate gradient for Newton steps — instead.

> **Zero gradient is not a minimum:** In high dimensions most critical points of a non-convex loss are saddles: one negative eigenvalue among millions suffices. Checking $\nabla f=0$ alone cannot tell them apart; the Hessian's smallest eigenvalue can.

> **Layout conventions differ:** This page uses the numerator layout ($J$ is $m\times n$, rows are output gradients). Some texts transpose it. Check shapes: $J\delta$ must have the shape of the output.

**Question:** $F:\mathbb{R}^3\to\mathbb{R}^2$. The shape of its Jacobian is…

- A. 2 × 3
- B. 3 × 2
- C. 3 × 3
- D. 2 × 2

**Answer:** A. 2 × 3 — One row per output ($m=2$), one column per input ($n=3$), so $J\delta\in\mathbb{R}^2$ for $\delta\in\mathbb{R}^3$.

**Question:** At a point with $\nabla f=0$, the Hessian has eigenvalues $4$ and $-1$. The point is…

- A. a local minimum
- B. a local maximum
- C. a saddle point
- D. impossible to classify

**Answer:** C. a saddle point — Mixed signs: $f$ curves up along the first eigenvector ($\tfrac12\cdot4\,t^2$) and down along the second ($-\tfrac12 t^2$). It is a minimum along one direction and a maximum along the other.

**Key points**

- $J_F$ ($m\times n$) is the local linear map: $F(x+\delta)\approx F(x)+J\delta$; $|\det J|$ scales volume.
- Jacobians multiply under composition — the chain rule and backprop.
- $H$ is symmetric; at a critical point its eigenvalue signs classify min, max, saddle or degenerate.
- Curvature sets the step size ($\eta<2/\lambda_{\max}$) and speed ($\kappa$); use $Hv$ products, not $H$, at scale.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
