# Derivatives & Gradients

> The gradient points in the direction of steepest increase; gradient descent steps the opposite way.

Source: https://www.mlfoundry.in/learn/derivatives-gradients  
Section: Mathematics › Calculus & Optimization  
Level: core · 6 min read · updated 2026-10-03

**The gradient:** of a differentiable function $f:\mathbb{R}^n\to\mathbb{R}$ at a point $x$ is the vector of its partial derivatives, $\nabla f(x)=\big(\partial f/\partial x_1,\dots,\partial f/\partial x_n\big)$. Each partial derivative is the rate of change of $f$ when only that one coordinate moves; together they give the rate of change in **every** direction.

- **Notation:** $f'(x)$, $\frac{df}{dx}$ in one variable; $\frac{\partial f}{\partial x_i}$, $\nabla f$, $\nabla_\theta L$ in several
- **Input → output:** a scalar function of $n$ variables → a vector field in $\mathbb{R}^n$
- **Key property:** $D_uf=\nabla f\cdot u$: the gradient is the steepest-ascent direction, perpendicular to level sets, and its length is the steepest slope
- **Prerequisites:** limits and slopes, vectors & the dot product
- **Used in:** gradient descent and every optimizer built on it, backprop, saliency maps, sensitivity analysis

## The intuition

In one variable the derivative $f'(x)$ is the slope of the tangent line: nudge $x$ by a tiny $h$ and $f$ changes by about $f'(x)\,h$. With $n$ variables you can nudge in infinitely many directions, but differentiability means the response is **linear** in the nudge — so $n$ numbers, one slope per axis, determine all of them. Those $n$ numbers are the gradient.

Picture $f$ as terrain and its level sets (contour lines) as a map. Walking along a contour keeps the height fixed, so the slope there is zero. The direction with the largest slope is the one at right angles to the contour, and that is where the gradient points. A loss function is just such terrain over parameter space, and training walks downhill on it.

## The math

$$
f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}{h},\qquad \frac{\partial f}{\partial x_i}(x)=\lim_{h\to0}\frac{f(x+h\,e_i)-f(x)}{h}
$$

- $h$: size of the nudge
- $e_i$: $i$-th standard basis vector (1 in slot $i$, 0 elsewhere)

Differentiability means $f$ is well approximated by a linear function near $x$: $f(x+\delta)=f(x)+\nabla f(x)\cdot\delta+o(\lVert\delta\rVert)$, where $o(\lVert\delta\rVert)$ shrinks faster than $\lVert\delta\rVert$. Setting $\delta=h\,u$ for a unit vector $u$ and dividing by $h$ gives the **directional derivative**:

$$
D_uf(x)=\lim_{h\to0}\frac{f(x+hu)-f(x)}{h}=\nabla f(x)\cdot u=\lVert\nabla f(x)\rVert\cos\theta
$$

- $u$: unit direction, $\lVert u\rVert=1$
- $\theta$: angle between $u$ and $\nabla f(x)$

Everything follows from the cosine. $D_uf$ is largest, equal to $\lVert\nabla f\rVert$, at $\theta=0$ — so $\nabla f$ is the direction of **steepest ascent** and its length is the steepest slope. It is most negative at $\theta=180°$ — so $-\nabla f$ is **steepest descent**. It is zero at $\theta=90°$; a curve inside a level set has $D_uf=0$ along its tangent, so the gradient is **perpendicular to level sets**. At a minimum, maximum or saddle, $\nabla f=0$.

**Gradient descent** turns this into an algorithm: repeatedly take a small step against the gradient. The first-order approximation predicts each step lowers $f$ by about $\eta\lVert\nabla f\rVert^2$.

$$
\theta_{t+1}=\theta_t-\eta\,\nabla L(\theta_t)
$$

- $\theta_t$: parameters after $t$ steps
- $\eta$: learning rate (step size), η > 0
- $L$: the loss being minimized

The step size has a hard limit. Along a direction where the loss has curvature (second derivative) $\lambda$, one step multiplies the distance to the minimum by $(1-\eta\lambda)$, which shrinks only if $0<\eta<2/\lambda$. The steepest direction sets the limit for all of them.

## See it

> Figure: A tilted bowl plus a Gaussian bump, on equal axes. At the starting point (−1.90, 0.35): ∂f/∂x = −0.610, ∂f/∂y = 0.315, ‖∇f‖ = 0.687, and the orange ∇f meets the green contour at a right angle. u starts 93° from ∇f, so D_u f = −0.030 — nearly zero, because u almost follows the contour. Press u ∥ ∇f and D_u f becomes exactly ‖∇f‖. The small arrows vanish at the minimum near (−0.64, −0.03).

> Figure: Gradient descent on f(x, y) = x² + 8y² from (−4.5, 2.2). The curvature along y is 16, so η past 1/16 makes y oscillate and η past 1/8 = 2/16 diverges. Click the plot to move the start.

## Worked example

$f(x,y)=x^2+3xy$ at $p=(1,2)$. Holding $y$ fixed, $\partial f/\partial x=2x+3y=8$; holding $x$ fixed, $\partial f/\partial y=3x=3$. So $\nabla f(p)=(8,3)$ and the steepest slope is $\lVert\nabla f\rVert=\sqrt{73}\approx8.544$.

In the direction $u=(0.6,0.8)$ the slope is $\nabla f\cdot u=4.8+2.4=7.2<8.544$, as the cosine formula requires. The direction $u=(3,-8)/\sqrt{73}$ is perpendicular to $\nabla f$ and has slope 0 — it is tangent to the level curve $f=7$ through $p$.

One gradient-descent step with $\eta=0.1$ moves to $p-0.1\,(8,3)=(0.2,1.7)$, where $f=0.04+1.02=1.06$, down from $f(p)=7$. The linear model predicted a drop of $\eta\lVert\nabla f\rVert^2=7.3$; the actual drop is $5.94$ — the step was large enough for curvature to matter.

## In code

```python
import numpy as np

f = lambda p: p[0]**2 + 3 * p[0] * p[1]
grad = lambda p: np.array([2 * p[0] + 3 * p[1], 3 * p[0]])

def num_grad(f, p, h=1e-5):              # central difference, one coordinate at a time
    E = np.eye(len(p))
    return np.array([(f(p + h * e) - f(p - h * e)) / (2 * h) for e in E])

p = np.array([1.0, 2.0])
print(grad(p), num_grad(f, p))            # [8. 3.] [8. 3.]

u = np.array([0.6, 0.8])
print(grad(p) @ u, np.linalg.norm(grad(p)))   # 7.2  8.544...  (no unit u beats ‖∇f‖)

p_new = p - 0.1 * grad(p)                 # one gradient-descent step, η = 0.1
print(f(p), f(p_new))                     # 7.0  1.06
```

Analytic gradient checked against central differences, the directional-derivative bound, and one descent step.

- **Training any model:** SGD, momentum and Adam all start from $\nabla_\theta L$ on a mini-batch; they differ only in how they turn it into a step.
- **Gradient checking:** Comparing a hand-written or autograd gradient with central differences catches backprop bugs before they silently degrade training.
- **Saliency & attribution:** $\partial(\text{class score})/\partial(\text{pixel})$ shows which input features the prediction is most sensitive to.
- **Adversarial examples:** FGSM perturbs an image by $\varepsilon\,\mathrm{sign}(\nabla_x L)$ — the steepest-ascent direction under the $L_\infty$ norm — to flip a classifier's prediction.

> **Steepest is not straight to the minimum:** $-\nabla f$ points across the contours, not toward the bottom. In an elongated bowl like $x^2+8y^2$ it points mostly along the steep $y$ axis, so descent zig-zags across the valley while crawling along it; the ratio of largest to smallest curvature (here 8) predicts how slow this gets. Feature standardization, momentum and Adam exist largely to fix this.

> **Zero gradient is not a minimum:** $\nabla f=0$ holds at minima, maxima and saddle points alike: $f(x,y)=x^2-y^2$ has $\nabla f(0,0)=0$ but decreases along $y$. Telling them apart needs second derivatives (the Hessian). In high-dimensional losses saddles far outnumber bad local minima.

**Question:** At some point $\nabla f=(3,4)$. Which unit direction $u$ gives $D_uf=0$?

- A. (0.6, 0.8)
- B. (0.8, −0.6)
- C. (−0.6, −0.8)
- D. (1, 0)

**Answer:** B. (0.8, −0.6) — $D_uf=\nabla f\cdot u=3(0.8)+4(-0.6)=0$: $u$ is perpendicular to the gradient, i.e. tangent to the level set. $(0.6,0.8)$ is the steepest-ascent direction with slope $\lVert\nabla f\rVert=5$, and $(-0.6,-0.8)$ gives $-5$.

**Question:** Gradient descent on $f(x)=5x^2$ converges from any start for every $\eta$ below which value?

- A. 0.1
- B. 0.2
- C. 0.5
- D. 1

**Answer:** B. 0.2 — $f'(x)=10x$, so a step gives $x-10\eta x=(1-10\eta)x$. The iterates shrink iff $|1-10\eta|<1$, i.e. $0<\eta<0.2=2/f''$.

**Key points**

- $\nabla f$ stacks the partial derivatives; $D_uf=\nabla f\cdot u=\lVert\nabla f\rVert\cos\theta$.
- The gradient is the steepest-ascent direction, its length the steepest slope, and it is perpendicular to level sets.
- Gradient descent steps $\theta\leftarrow\theta-\eta\nabla L$; stability needs $\eta<2/\lambda_{\max}$.
- $\nabla f=0$ marks minima, maxima and saddles alike; ill-conditioning makes descent zig-zag.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
