Sharpness-aware minimization (SAM)

Minimize the worst loss in a ρ-ball around the weights, not the loss at a point. One extra gradient step to the approximate worst case, ε ≈ ρ g/‖g‖, steers training toward flat minima that generalize better, for about twice the compute.

12 min read1007 wordsadvanced0 viewsupdated 2026-10-11

Definition

Sharpness-aware minimization (SAM; Foret, Kleiner, Mobahi and Neyshabur, ICLR 2021) trains on the min-max objective min⁡wmax⁡∥ϵ∥2≤ρL(w+ϵ)\min_w\max_{\|\epsilon\|_2\le\rho}L(w+\epsilon): the loss at the worst point within distance ρ\rho of the weights. Each step first moves the weights uphill to w+ϵ^w+\hat\epsilon, ϵ^=ρ g/∥g∥\hat\epsilon=\rho\,g/\|g\|, takes the gradient there, and then applies that gradient at the original ww.

Over-parameterized networks have many weight settings with near-zero training loss, and they do not all generalize equally well. A long-standing observation is that flat minima, where the loss stays low in a whole neighbourhood, tend to generalize better than sharp ones, where a small change of the weights raises the loss steeply. A rough intuition: the test loss surface is a slightly shifted copy of the training surface, and a shift costs little in a wide basin and a lot in a narrow one.

Hochreiter and Schmidhuber ("Flat minima", 1997) proposed searching for flat regions explicitly. Keskar et al. (ICLR 2017) linked the generalization gap of large-batch training to convergence to sharp minima. Dinh et al. (ICML 2017) pointed out a caveat: for ReLU networks you can rescale weights to make any minimum look arbitrarily sharp without changing the function, so sharpness must be measured carefully. Jiang et al. (ICLR 2020), comparing dozens of generalization measures, still found sharpness-based ones among the most predictive. SAM turned this idea into a simple, scalable optimizer. It improved results on CIFAR-10/100 and ImageNet, made training robust to label noise, and let vision transformers and MLP-Mixers trained from scratch on ImageNet match or beat ResNets of similar size (Chen, Hsieh and Gong, ICLR 2022).

The objective and its approximation

min⁡w  LSAM(w)+λ∥w∥22,LSAM(w)=max⁡∥ϵ∥2≤ρL(w+ϵ)=L(w)+[max⁡∥ϵ∥2≤ρL(w+ϵ)−L(w)]⏟sharpness\min_w\; L^{\text{SAM}}(w)+\lambda\|w\|_2^2,\qquad L^{\text{SAM}}(w)=\max_{\|\epsilon\|_2\le\rho}L(w+\epsilon)=L(w)+\underbrace{\Big[\max_{\|\epsilon\|_2\le\rho}L(w+\epsilon)-L(w)\Big]}_{\text{sharpness}}
ρ\rho
neighbourhood radius; Foret et al. use ρ=0.05\rho=0.05 for most CIFAR and ImageNet runs
ϵ\epsilon
an adversarial weight perturbation, not an input perturbation
λ∥w∥2\lambda\|w\|^2
ordinary weight decay, kept as usual

The inner maximization is intractable exactly, so SAM linearizes: L(w+ϵ)≈L(w)+ϵ⊤gL(w+\epsilon)\approx L(w)+\epsilon^\top g with g=∇L(w)g=\nabla L(w). Maximizing a linear function over a ball puts ϵ\epsilon on the boundary, in the direction of the gradient:

ϵ^(w)=arg⁡max⁡∥ϵ∥2≤ρϵ⊤g=ρ g∥g∥2,∇wLSAM(w)≈∇L(w)∣w+ϵ^(w)\hat\epsilon(w)=\arg\max_{\|\epsilon\|_2\le\rho}\epsilon^\top g=\rho\,\frac{g}{\|g\|_2},\qquad \nabla_w L^{\text{SAM}}(w)\approx\nabla L(w)\big|_{w+\hat\epsilon(w)}
ϵ^\hat\epsilon
one normalized gradient-ascent step of length ρ\rho: the worst direction to first order. For a general ℓp\ell_p ball it is ρ sign⁡(g)∣g∣q−1/∥g∥qq−1\rho\,\operatorname{sign}(g)|g|^{q-1}/\|g\|_q^{q-1} with 1/p+1/q=11/p+1/q=1
∇L∣w+ϵ^\nabla L|_{w+\hat\epsilon}
the gradient evaluated at the perturbed point. The exact gradient also has a term through dϵ^/dwd\hat\epsilon/dw, which involves the Hessian; SAM drops it, and Foret et al. found that keeping it made results worse

One SAM step (two forward-backward passes)

  1. 1Pass 1: compute g=∇LB(w)g=\nabla L_B(w) on the mini-batch BB.
  2. 2Perturb: ϵ^=ρ g/∥g∥\hat\epsilon=\rho\,g/\|g\|, with the norm taken over all parameters together, and move to w+ϵ^w+\hat\epsilon.
  3. 3Pass 2: compute gSAM=∇LB(w+ϵ^)g_{\text{SAM}}=\nabla L_B(w+\hat\epsilon) on the same batch.
  4. 4Restore ww (subtract ϵ^\hat\epsilon) and give gSAMg_{\text{SAM}} to the base optimizer (SGD with momentum, AdamW, …): w←BaseStep(w, gSAM)w\leftarrow\text{BaseStep}(w,\,g_{\text{SAM}}).
L = ½(1 + 2u²)·v² · every point on v = 0 is a minimumstartflat (λmax = 1)sharp━ GD ━ SAM ◌ ε-ball ━ ε = ρ g/‖g‖ → −∇L(w+ε) ━ floor sharpnesstest loss = ½a(u)(v − 0.3)²: at a minimum it is ½a(u)·0.3², so sharper means worsetrain loss (log) vs step1e-61e-41e-21sharpness λmax(H) vs step048120100200
step 0
● GD u 2.200 · train 12.015 · λmax 21.15 · test 7.690 · gradient evals 0
● SAM u 2.200 · train 12.015 · λmax 21.15 · test 7.690 · gradient evals 0
fig.Every point on the floor v = 0 of L = ½(1 + 2u²)v² is a global minimum with zero training loss, but the curvature across the floor, λmax = 1 + 2u², is 1 in the middle and grows toward the ends. The test loss is the same surface shifted by 0.3, so at a minimum it equals 0.045·λmax. From the start (2.2, 1.5), GD at lr 0.05 drops straight onto the floor and stops at u = 1.556, λmax = 5.84, test loss 0.263. SAM with ρ = 0.3 reaches the floor too, then bounces across it and drifts along it: after 200 steps it is at u = 0.187, λmax = 1.07, test loss 0.051, with a training loss of 3.6e-5 (not 0, because it keeps bouncing) and twice as many gradient evaluations (400). The ring is the ρ-ball, the violet segment is ε̂, and the orange arrow is −∇L(w + ε̂), the direction SAM actually steps in. ρ = 0.1 stops at λmax 3.68; ρ = 0.5 reaches 1.00.

Why SAM drifts toward flat regions

Near a minimum the gradient is small, so the gradient-ascent step ϵ^\hat\epsilon points across the valley, along the sharpest direction, and L(w+ϵ^)≈L(w)+12ρ2λmax⁡L(w+\hat\epsilon)\approx L(w)+\tfrac12\rho^2\lambda_{\max}. Its gradient therefore contains a term 12ρ2∇λmax⁡\tfrac12\rho^2\nabla\lambda_{\max}, which pushes along the valley floor toward lower curvature, even though the training loss itself is already flat in that direction. Wen, Ma and Li ("How does sharpness-aware minimization minimize sharpness?", 2022) proved that full-batch SAM near a manifold of minima does follow this flow and minimizes λmax⁡\lambda_{\max}; Bartlett, Long and Bousquet (2023) describe the same dynamics as "bouncing across ravines and drifting towards wide minima". That is the picture in the figure.

Worked example: GD vs SAM gradient near the floor

In the figure's surface L(u,v)=12(1+2u2)v2L(u,v)=\tfrac12(1+2u^2)v^2 the gradient is ∇L=(2uv2,  (1+2u2)v)\nabla L=(2uv^2,\;(1+2u^2)v). Take the point (u,v)=(1, 0.01)(u,v)=(1,\,0.01), almost on the floor, with ρ=0.3\rho=0.3.

quantityGD at wSAM at w + ε̂
evaluation point(1, 0.01)(1.0020, 0.3100)
loss1.5×10−41.5\times10^{-4}0.1445
gradient u-component (along the floor)0.00020.1926
gradient v-component (across the floor)0.03000.9325
Computed with the figure's code. GD barely moves along the floor; SAM's u-gradient is about 960 times larger.
  1. 1g=(0.0002,  0.03)g=(0.0002,\;0.03), ∥g∥=0.03\|g\|=0.03, so ϵ^=0.3 g/∥g∥=(0.0020,  0.2999)\hat\epsilon=0.3\,g/\|g\|=(0.0020,\;0.2999): almost straight across the valley.
  2. 2At w+ϵ^=(1.002, 0.310)w+\hat\epsilon=(1.002,\,0.310), ∂L/∂u=2⋅1.002⋅0.3102=0.1926\partial L/\partial u=2\cdot1.002\cdot0.310^2=0.1926. To first order this is 12ρ2 ∂λmax⁡/∂u=12⋅0.09⋅4u=0.18\tfrac12\rho^2\,\partial\lambda_{\max}/\partial u=\tfrac12\cdot0.09\cdot4u=0.18.
  3. 3The vv-component 0.930.93 overshoots the floor (from v=0.01v=0.01 to v≈−0.036v\approx-0.036 at η=0.05\eta=0.05), so the next ϵ^\hat\epsilon points the other way: SAM zigzags across the floor while the uu-component, which has the same sign on both sides, accumulates.
  4. 4At the start (2.2, 1.5)(2.2,\,1.5), far from the floor: L=12.0L=12.0, ϵ^=(0.158, 0.255)\hat\epsilon=(0.158,\,0.255), L(w+ϵ^)=18.7L(w+\hat\epsilon)=18.7 and gSAM=(14.5, 21.3)g_{\text{SAM}}=(14.5,\,21.3) vs g=(9.9, 16.0)g=(9.9,\,16.0). Here SAM behaves like GD with a somewhat larger gradient.
python
1import torch
2
3def sam_step(model, loss_fn, x, y, base_opt, rho=0.05, adaptive=False):
4 """One SAM (or ASAM if adaptive=True) update. Returns the loss at w."""
5 params = [p for p in model.parameters() if p.requires_grad]
6
7 # pass 1: gradient at w
8 loss = loss_fn(model(x), y)
9 loss.backward()
10
11 with torch.no_grad():
12 # ε̂ = ρ g/‖g‖ (SAM) or ρ T²g/‖Tg‖ with T = |w| (ASAM); norm over ALL parameters
13 scaled = [(p.abs() * p.grad) if adaptive else p.grad for p in params if p.grad is not None]
14 norm = torch.norm(torch.stack([s.norm(2) for s in scaled]), 2)
15 eps = []
16 for p in params:
17 if p.grad is None:
18 eps.append(None)
19 continue
20 e = (p.pow(2) if adaptive else 1.0) * p.grad * (rho / (norm + 1e-12))
21 p.add_(e) # climb to w + ε̂
22 eps.append(e)
23 base_opt.zero_grad()
24
25 # pass 2: gradient at w + ε̂, same batch
26 loss_fn(model(x), y).backward()
27
28 with torch.no_grad():
29 for p, e in zip(params, eps):
30 if e is not None:
31 p.sub_(e) # back to w
32 base_opt.step() # step from w with the SAM gradient
33 base_opt.zero_grad()
34 return loss.detach()
35
36model = torch.nn.Sequential(torch.nn.Linear(20, 64), torch.nn.ReLU(), torch.nn.Linear(64, 10))
37base = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9, weight_decay=5e-4)
38loss_fn = torch.nn.CrossEntropyLoss()
39for x, y in [(torch.randn(128, 20), torch.randint(0, 10, (128,)))] * 100:
40 sam_step(model, loss_fn, x, y, base, rho=0.05)
PyTorch has no built-in SAM optimizer: wrap any base optimizer with two forward-backward passes

Variants: m-sharpness, ASAM and cheaper SAMs

m-sharpness. In data-parallel training each worker can compute its own ϵ^\hat\epsilon from its own shard of mm examples, without synchronizing the first pass. Foret et al. found that this per-shard version generalizes better than computing ϵ^\hat\epsilon from the whole batch, and that smaller mm helped; Andriushchenko and Flammarion (ICML 2022) studied why. So the practical SAM is really "mm-SAM", and the unsynchronized version is also cheaper.

ASAM (Kwon, Kim, Park and Choi, ICML 2021) answers the reparameterization critique. It measures the neighbourhood relative to each weight's size, ∥Tw−1ϵ∥≤ρ\|T_w^{-1}\epsilon\|\le\rho with Tw=diag⁡(∣w∣)T_w=\operatorname{diag}(|w|), so rescaling a layer no longer changes the measured sharpness. The perturbation becomes ϵ^=ρ Tw2g/∥Twg∥\hat\epsilon=\rho\,T_w^2g/\|T_wg\| (the adaptive=True branch in the code), and ρ\rho is typically an order of magnitude larger than SAM's.

Cost. Two forward-backward passes per step double the compute. Cheaper variants reuse the perturbation for several steps (LookSAM), perturb only a subset of weights or examples (ESAM), or schedule ρ\rho; GSAM adds a surrogate gap term. In practice people also compare SAM against plain training run for twice as many epochs, the fair compute-matched baseline.

Use when

  • +Consistent generalization gains on image classification, especially for ViTs and MLP-Mixers trained from scratch
  • +More robust to label noise
  • +Wraps any base optimizer; one extra hyperparameter, ρ
  • +Explicit, tunable bias toward low-curvature solutions

Avoid when

  • −Two forward-backward passes: about twice the compute per step
  • −Sharpness is not reparameterization-invariant (motivates ASAM)
  • −Extra care with BatchNorm statistics, mixed precision and distributed gradients
  • −Gains are smaller or absent on some tasks, and should be compared against training 2× longer

Check yourself

In SAM, at which weights is the gradient that the base optimizer finally uses evaluated, and where is the update applied?

Check yourself

Why is the SAM perturbation ε̂ = ρ g/‖g‖?

Remember

  • ✓SAM minimizes max⁡∥ϵ∥≤ρL(w+ϵ)\max_{\|\epsilon\|\le\rho}L(w+\epsilon): training loss plus a sharpness penalty.
  • ✓First-order solution: ϵ^=ρ g/∥g∥\hat\epsilon=\rho\,g/\|g\|; use ∇L(w+ϵ^)\nabla L(w+\hat\epsilon) as the gradient at ww, dropping the Hessian term.
  • ✓Two forward-backward passes per step on the same batch; wrap any base optimizer. PyTorch has no built-in SAM.
  • ✓Near a valley of minima SAM bounces across and drifts along the floor toward lower λmax⁡\lambda_{\max}, the flat minima that tend to generalize better.
  • ✓Variants: m-sharpness (per-shard ϵ^\hat\epsilon, often better), ASAM (scale-invariant, Tw=∣w∣T_w=|w|), and cheaper LookSAM/ESAM; typical ρ=0.05\rho=0.05.
← back to Deep Learning roadmap