Lagrange Multipliers

At a constrained optimum, the gradient of the objective is parallel to the gradient of the constraint: ∇f = λ∇g.

6 min read608 wordsadvanced3 viewsupdated 2026-10-03

Definition

Lagrange multiplier To optimize f(x)f(x) subject to g(x)=0g(x)=0, introduce a scalar λ\lambda and look for points where ∇f(x)=λ∇g(x)\nabla f(x)=\lambda\nabla g(x) and g(x)=0g(x)=0. The Lagrange multiplier λ\lambda is the factor between the two gradients; at the optimum it equals the rate at which the best achievable ff changes as the constraint is relaxed.

At a glance

Notation
Lagrangian L(x,λ)=f(x)−λ g(x)\mathcal{L}(x,\lambda)=f(x)-\lambda\,g(x); several constraints: f−∑iλigif-\sum_i\lambda_i g_i
Input → output
objective ff and constraints gi=0g_i=0 → candidate points x∗x^* with multipliers λ∗\lambda^*
Key property
∇f(x∗)=λ∗∇g(x∗)\nabla f(x^*)=\lambda^*\nabla g(x^*), and λ∗=df∗dc\lambda^*=\dfrac{d f^*}{dc} for the constraint g(x)=cg(x)=c
Inequalities
KKT conditions: μ≥0\mu\ge0 and μ g(x)=0\mu\,g(x)=0 (complementary slackness)
Prerequisites
derivatives & gradients, Jacobian, convexity
Used in
PCA, SVM duals, maximum entropy / softmax, regularization, constrained resource allocation

The intuition

Walk along the constraint curve and watch ff. At a constrained maximum you can't go higher by moving along the curve, so ff doesn't change to first order in the direction the curve runs: ∇f\nabla f has no component along the curve. The only direction left is perpendicular to the curve — and that is the direction of ∇g\nabla g, which is always perpendicular to its level set. So the two gradients line up.

Geometrically, the optimum is where a contour of ff just touches the constraint curve without crossing it. Crossing would mean higher and lower values of ff on either side, so you could still improve.

The math

L(x,λ)=f(x)−λ g(x),∇xL=∇f(x)−λ∇g(x)=0,∂L∂λ=−g(x)=0\mathcal{L}(x,\lambda)=f(x)-\lambda\,g(x),\qquad \nabla_x\mathcal{L}=\nabla f(x)-\lambda\nabla g(x)=0,\qquad \frac{\partial\mathcal{L}}{\partial\lambda}=-g(x)=0
ff
objective to maximize or minimize
g(x)=0g(x)=0
equality constraint; the feasible set is its level set
λ\lambda
Lagrange multiplier, one per constraint

Why parallel. Let tt be any direction tangent to the constraint at x∗x^*, so ∇g(x∗)⊤t=0\nabla g(x^*)^\top t=0. Moving along the constraint changes ff at rate ∇f(x∗)⊤t\nabla f(x^*)^\top t, which must be zero at an optimum. So ∇f\nabla f is orthogonal to every tangent direction, i.e. it lies in the span of ∇g\nabla g. With kk constraints the same argument gives ∇f=∑iλi∇gi\nabla f=\sum_i\lambda_i\nabla g_i, valid when the ∇gi\nabla g_i are linearly independent at x∗x^* (the constraint qualification).

Meaning of λ. For the family g(x)=cg(x)=c, let f∗(c)f^*(c) be the optimal value. Then df∗/dc=λ∗df^*/dc=\lambda^*: the multiplier is the marginal value, or shadow price, of the constraint. λ∗=0\lambda^*=0 means the constraint isn't binding.

Inequality constraints (KKT). For min⁡f\min f subject to h(x)≤0h(x)\le0, write ∇f+μ∇h=0\nabla f+\mu\nabla h=0 with μ≥0\mu\ge0, plus complementary slackness μ h(x)=0\mu\,h(x)=0: either the constraint is active (h=0h=0) and pushes back with μ>0\mu>0, or it is slack and μ=0\mu=0. For convex problems these conditions are sufficient as well as necessary.

Solving an equality-constrained problem

  1. 1Write L=f−∑iλigi\mathcal{L}=f-\sum_i\lambda_i g_i.
  2. 2Set ∇xL=0\nabla_x\mathcal{L}=0 and gi(x)=0g_i(x)=0: n+kn+k equations in n+kn+k unknowns.
  3. 3Solve for all candidates (x,λ)(x,\lambda); also check points where the ∇gi\nabla g_i are dependent.
  4. 4Evaluate ff at every candidate (or on a compact feasible set, compare them) to pick the max and the min.

See it

g = x² + y² − 1 = 0max f = 1.414min f = −1.414∇f∇gcontours of f = x + y · orange f > 0, blue f < 0

point (-0.866, 0.500) · φ = 150°

f = x + y = -0.366

angle(∇f, ∇g) = 105.0°

along-circle part of ∇f = -1.366

best λ = ∇f·∇g / ‖∇g‖² = -0.183

Not optimal: move along the violet arrow to increase f while staying on the circle.

drag the white point around the circle · it snaps to the optima within 2°

fig.Optimize f = x + y on the unit circle. At the start (150°) ∇f and ∇g are 105° apart and ∇f has an along-circle part (violet) of −1.366: moving along the circle still changes f. Drag to 45°: the angle reaches 0°, ∇f = 0.707·∇g, and f = 1.414 is the maximum. At 225° they are antiparallel, λ = −0.707, f = −1.414 is the minimum.

Worked example

Minimize f(x,y)=x2+2y2f(x,y)=x^2+2y^2 subject to x+y=3x+y=3. Stationarity of L=x2+2y2−λ(x+y−3)\mathcal{L}=x^2+2y^2-\lambda(x+y-3) gives 2x=λ2x=\lambda and 4y=λ4y=\lambda, so x=2yx=2y; the constraint then gives y=1y=1, x=2x=2, λ=4\lambda=4, and f∗=6f^*=6.

Check the shadow price. With constraint x+y=cx+y=c, the same steps give x=2c3x=\tfrac{2c}{3}, y=c3y=\tfrac{c}{3} and f∗(c)=23c2f^*(c)=\tfrac{2}{3}c^2, so df∗/dc=43c=4df^*/dc=\tfrac43c=4 at c=3c=3 — exactly λ\lambda. Raising cc to 3.13.1 raises the minimum to about 6.46.4 (exactly 6.4076.407).

The viz problem. For f=x+yf=x+y on x2+y2=1x^2+y^2=1: (1,1)=λ(2x,2y)(1,1)=\lambda(2x,2y) gives x=y=12λx=y=\tfrac{1}{2\lambda}, and the constraint gives λ=±12=±0.707\lambda=\pm\tfrac{1}{\sqrt2}=\pm0.707. The two candidates are the maximum f=2f=\sqrt2 and the minimum f=−2f=-\sqrt2 — the method finds both.

In code

python
1import numpy as np
2from scipy.optimize import minimize
3
4f = lambda v: v[0]**2 + 2 * v[1]**2
5con = {"type": "eq", "fun": lambda v: v[0] + v[1] - 3}
6res = minimize(f, x0=[0.0, 0.0], method="SLSQP", constraints=[con])
7res.x.round(4), round(res.fun, 4) # (array([2., 1.]), 6.0)
8
9# PCA: grad(v^T S v) = lambda * grad(v^T v) => S v = lambda v
10S = np.array([[2.0, 1.0], [1.0, 2.0]])
11lam, V = np.linalg.eigh(S)
12lam[-1], V[:, -1] # 3.0, ±[0.707, 0.707]: max variance = λ
SLSQP handles equality and inequality constraints directly. PCA is the Lagrange problem max vᵀΣv subject to ‖v‖² = 1, whose stationarity condition Σv = λv is the eigen-equation.

Where it's used

PCA

Maximizing variance v⊤Σvv^\top\Sigma v subject to ∥v∥=1\|v\|=1 gives Σv=λv\Sigma v=\lambda v: the principal directions are eigenvectors, and each multiplier λ\lambda is the variance captured.

Support vector machines

The SVM dual comes from the Lagrangian of the margin constraints; complementary slackness makes αi>0\alpha_i>0 only for points on or inside the margin — the support vectors.

Maximum entropy and softmax

Maximizing entropy subject to ∑kpk=1\sum_k p_k=1 and a fixed expected score gives pk∝eβskp_k\propto e^{\beta s_k}: softmax, with the inverse temperature β\beta as the multiplier on the score constraint.

Regularization

Ridge regression minimizes squared error subject to ∥w∥2≤t\|w\|^2\le t; its Lagrangian is the familiar penalized loss, and the penalty strength is the KKT multiplier for that budget.

Check yourself

Maximize f=xyf=xy subject to x+y=10x+y=10. The optimum and multiplier are…

Check yourself

At the optimum of min⁡f\min f subject to g(x)=cg(x)=c, the multiplier is λ∗=4\lambda^*=4. If cc increases by 0.10.1, the optimal value changes by about…

Remember

  • ✓At a constrained optimum ∇f=λ∇g\nabla f=\lambda\nabla g: the contour of ff is tangent to the constraint.
  • ✓Solve ∇xL=0\nabla_x\mathcal{L}=0 and g=0g=0 together; compare all candidates.
  • ✓λ=df∗/dc\lambda=df^*/dc — the shadow price of the constraint; λ=0\lambda=0 means it isn't binding.
  • ✓KKT extends this to inequalities with μ ≥ 0 and complementary slackness — the basis of SVMs.
← back to Mathematics roadmap