# Lagrange Multipliers

> At a constrained optimum, the gradient of the objective is parallel to the gradient of the constraint: ∇f = λ∇g.

Source: https://www.mlfoundry.in/learn/lagrange-multipliers  
Section: Mathematics › Calculus & Optimization  
Level: advanced · 6 min read · updated 2026-10-03

**Lagrange multiplier:** To optimize $f(x)$ subject to $g(x)=0$, introduce a scalar $\lambda$ and look for points where $\nabla f(x)=\lambda\nabla g(x)$ and $g(x)=0$. The **Lagrange multiplier** $\lambda$ is the factor between the two gradients; at the optimum it equals the rate at which the best achievable $f$ changes as the constraint is relaxed.

- **Notation:** Lagrangian $\mathcal{L}(x,\lambda)=f(x)-\lambda\,g(x)$; several constraints: $f-\sum_i\lambda_i g_i$
- **Input → output:** objective $f$ and constraints $g_i=0$ → candidate points $x^*$ with multipliers $\lambda^*$
- **Key property:** $\nabla f(x^*)=\lambda^*\nabla g(x^*)$, and $\lambda^*=\dfrac{d f^*}{dc}$ for the constraint $g(x)=c$
- **Inequalities:** KKT conditions: $\mu\ge0$ and $\mu\,g(x)=0$ (complementary slackness)
- **Prerequisites:** derivatives & gradients, Jacobian, convexity
- **Used in:** PCA, SVM duals, maximum entropy / softmax, regularization, constrained resource allocation

## The intuition

Walk along the constraint curve and watch $f$. At a constrained maximum you can't go higher by moving along the curve, so $f$ doesn't change to first order in the direction the curve runs: $\nabla f$ has no component along the curve. The only direction left is perpendicular to the curve — and that is the direction of $\nabla g$, which is always perpendicular to its level set. So the two gradients line up.

Geometrically, the optimum is where a contour of $f$ just touches the constraint curve without crossing it. Crossing would mean higher and lower values of $f$ on either side, so you could still improve.

## The math

$$
\mathcal{L}(x,\lambda)=f(x)-\lambda\,g(x),\qquad \nabla_x\mathcal{L}=\nabla f(x)-\lambda\nabla g(x)=0,\qquad \frac{\partial\mathcal{L}}{\partial\lambda}=-g(x)=0
$$

- $f$: objective to maximize or minimize
- $g(x)=0$: equality constraint; the feasible set is its level set
- $\lambda$: Lagrange multiplier, one per constraint

**Why parallel.** Let $t$ be any direction tangent to the constraint at $x^*$, so $\nabla g(x^*)^\top t=0$. Moving along the constraint changes $f$ at rate $\nabla f(x^*)^\top t$, which must be zero at an optimum. So $\nabla f$ is orthogonal to every tangent direction, i.e. it lies in the span of $\nabla g$. With $k$ constraints the same argument gives $\nabla f=\sum_i\lambda_i\nabla g_i$, valid when the $\nabla g_i$ are linearly independent at $x^*$ (the *constraint qualification*).

**Meaning of λ.** For the family $g(x)=c$, let $f^*(c)$ be the optimal value. Then $df^*/dc=\lambda^*$: the multiplier is the marginal value, or *shadow price*, of the constraint. $\lambda^*=0$ means the constraint isn't binding.

**Inequality constraints (KKT).** For $\min f$ subject to $h(x)\le0$, write $\nabla f+\mu\nabla h=0$ with $\mu\ge0$, plus **complementary slackness** $\mu\,h(x)=0$: either the constraint is active ($h=0$) and pushes back with $\mu>0$, or it is slack and $\mu=0$. For convex problems these conditions are sufficient as well as necessary.

**Solving an equality-constrained problem**

1. Write $\mathcal{L}=f-\sum_i\lambda_i g_i$.
2. Set $\nabla_x\mathcal{L}=0$ and $g_i(x)=0$: $n+k$ equations in $n+k$ unknowns.
3. Solve for all candidates $(x,\lambda)$; also check points where the $\nabla g_i$ are dependent.
4. Evaluate $f$ at every candidate (or on a compact feasible set, compare them) to pick the max and the min.

## See it

> Figure: Optimize f = x + y on the unit circle. At the start (150°) ∇f and ∇g are 105° apart and ∇f has an along-circle part (violet) of −1.366: moving along the circle still changes f. Drag to 45°: the angle reaches 0°, ∇f = 0.707·∇g, and f = 1.414 is the maximum. At 225° they are antiparallel, λ = −0.707, f = −1.414 is the minimum.

## Worked example

Minimize $f(x,y)=x^2+2y^2$ subject to $x+y=3$. Stationarity of $\mathcal{L}=x^2+2y^2-\lambda(x+y-3)$ gives $2x=\lambda$ and $4y=\lambda$, so $x=2y$; the constraint then gives $y=1$, $x=2$, $\lambda=4$, and $f^*=6$.

**Check the shadow price.** With constraint $x+y=c$, the same steps give $x=\tfrac{2c}{3}$, $y=\tfrac{c}{3}$ and $f^*(c)=\tfrac{2}{3}c^2$, so $df^*/dc=\tfrac43c=4$ at $c=3$ — exactly $\lambda$. Raising $c$ to $3.1$ raises the minimum to about $6.4$ (exactly $6.407$).

**The viz problem.** For $f=x+y$ on $x^2+y^2=1$: $(1,1)=\lambda(2x,2y)$ gives $x=y=\tfrac{1}{2\lambda}$, and the constraint gives $\lambda=\pm\tfrac{1}{\sqrt2}=\pm0.707$. The two candidates are the maximum $f=\sqrt2$ and the minimum $f=-\sqrt2$ — the method finds both.

## In code

```python
import numpy as np
from scipy.optimize import minimize

f = lambda v: v[0]**2 + 2 * v[1]**2
con = {"type": "eq", "fun": lambda v: v[0] + v[1] - 3}
res = minimize(f, x0=[0.0, 0.0], method="SLSQP", constraints=[con])
res.x.round(4), round(res.fun, 4)        # (array([2., 1.]), 6.0)

# PCA: grad(v^T S v) = lambda * grad(v^T v)  =>  S v = lambda v
S = np.array([[2.0, 1.0], [1.0, 2.0]])
lam, V = np.linalg.eigh(S)
lam[-1], V[:, -1]                        # 3.0, ±[0.707, 0.707]: max variance = λ
```

SLSQP handles equality and inequality constraints directly. PCA is the Lagrange problem max vᵀΣv subject to ‖v‖² = 1, whose stationarity condition Σv = λv is the eigen-equation.

- **PCA:** Maximizing variance $v^\top\Sigma v$ subject to $\|v\|=1$ gives $\Sigma v=\lambda v$: the principal directions are eigenvectors, and each multiplier $\lambda$ *is* the variance captured.
- **Support vector machines:** The SVM dual comes from the Lagrangian of the margin constraints; complementary slackness makes $\alpha_i>0$ only for points on or inside the margin — the support vectors.
- **Maximum entropy and softmax:** Maximizing entropy subject to $\sum_k p_k=1$ and a fixed expected score gives $p_k\propto e^{\beta s_k}$: softmax, with the inverse temperature $\beta$ as the multiplier on the score constraint.
- **Regularization:** Ridge regression minimizes squared error subject to $\|w\|^2\le t$; its Lagrangian is the familiar penalized loss, and the penalty strength is the KKT multiplier for that budget.

> **The conditions find candidates, not answers:** $\nabla f=\lambda\nabla g$ holds at constrained maxima, minima and saddles alike — the circle example returns both $\pm\sqrt2$. Evaluate $f$ at every candidate, or use convexity or second-order conditions, before calling one the optimum.

> **When ∇g = 0 the method can miss the optimum:** Minimize $x$ subject to $y^2-x^3=0$. Feasible points have $x\ge0$, so the minimum is $(0,0)$ — but there $\nabla g=(-3x^2,2y)=(0,0)$ and $\nabla f=(1,0)\ne\lambda\cdot0$. No multiplier exists. Always check points where the constraint qualification fails.

> **Sign conventions vary:** Texts write $f-\lambda g$ or $f+\lambda g$; the second flips the sign of $\lambda$. The geometry — parallel gradients — and $|\lambda|$ are the same. With $f-\lambda g$ and $g(x)=c$, $\lambda=df^*/dc$.

**Question:** Maximize $f=xy$ subject to $x+y=10$. The optimum and multiplier are…

- A. x = y = 5, λ = 5
- B. x = y = 5, λ = 25
- C. x = 10, y = 0, λ = 0
- D. x = y = 5, λ = 1

**Answer:** A. x = y = 5, λ = 5 — $\nabla f=(y,x)=\lambda(1,1)$ gives $x=y=\lambda$; the constraint gives $x=y=5$, so $\lambda=5$ and $f^*=25$. Check: $f^*(c)=c^2/4$, so $df^*/dc=c/2=5$.

**Question:** At the optimum of $\min f$ subject to $g(x)=c$, the multiplier is $\lambda^*=4$. If $c$ increases by $0.1$, the optimal value changes by about…

- A. +0.4
- B. −0.4
- C. +4
- D. 0 — λ only matters for inequality constraints

**Answer:** A. +0.4 — $df^*/dc=\lambda^*$, so $\Delta f^*\approx\lambda^*\,\Delta c=4\times0.1=0.4$. In the worked example $f^*$ goes from 6 to 6.407.

**Key points**

- At a constrained optimum $\nabla f=\lambda\nabla g$: the contour of $f$ is tangent to the constraint.
- Solve $\nabla_x\mathcal{L}=0$ and $g=0$ together; compare all candidates.
- $\lambda=df^*/dc$ — the shadow price of the constraint; $\lambda=0$ means it isn't binding.
- KKT extends this to inequalities with μ ≥ 0 and complementary slackness — the basis of SVMs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
