Definition
Sharpness-aware minimization (SAM; Foret, Kleiner, Mobahi and Neyshabur, ICLR 2021) trains on the min-max objective : the loss at the worst point within distance of the weights. Each step first moves the weights uphill to , , takes the gradient there, and then applies that gradient at the original .
Over-parameterized networks have many weight settings with near-zero training loss, and they do not all generalize equally well. A long-standing observation is that flat minima, where the loss stays low in a whole neighbourhood, tend to generalize better than sharp ones, where a small change of the weights raises the loss steeply. A rough intuition: the test loss surface is a slightly shifted copy of the training surface, and a shift costs little in a wide basin and a lot in a narrow one.
Hochreiter and Schmidhuber ("Flat minima", 1997) proposed searching for flat regions explicitly. Keskar et al. (ICLR 2017) linked the generalization gap of large-batch training to convergence to sharp minima. Dinh et al. (ICML 2017) pointed out a caveat: for ReLU networks you can rescale weights to make any minimum look arbitrarily sharp without changing the function, so sharpness must be measured carefully. Jiang et al. (ICLR 2020), comparing dozens of generalization measures, still found sharpness-based ones among the most predictive. SAM turned this idea into a simple, scalable optimizer. It improved results on CIFAR-10/100 and ImageNet, made training robust to label noise, and let vision transformers and MLP-Mixers trained from scratch on ImageNet match or beat ResNets of similar size (Chen, Hsieh and Gong, ICLR 2022).
The objective and its approximation
- neighbourhood radius; Foret et al. use for most CIFAR and ImageNet runs
- an adversarial weight perturbation, not an input perturbation
- ordinary weight decay, kept as usual
The inner maximization is intractable exactly, so SAM linearizes: with . Maximizing a linear function over a ball puts on the boundary, in the direction of the gradient:
- one normalized gradient-ascent step of length : the worst direction to first order. For a general ball it is with
- the gradient evaluated at the perturbed point. The exact gradient also has a term through , which involves the Hessian; SAM drops it, and Foret et al. found that keeping it made results worse
One SAM step (two forward-backward passes)
- 1Pass 1: compute on the mini-batch .
- 2Perturb: , with the norm taken over all parameters together, and move to .
- 3Pass 2: compute on the same batch.
- 4Restore (subtract ) and give to the base optimizer (SGD with momentum, AdamW, …): .
Why SAM drifts toward flat regions
Near a minimum the gradient is small, so the gradient-ascent step points across the valley, along the sharpest direction, and . Its gradient therefore contains a term , which pushes along the valley floor toward lower curvature, even though the training loss itself is already flat in that direction. Wen, Ma and Li ("How does sharpness-aware minimization minimize sharpness?", 2022) proved that full-batch SAM near a manifold of minima does follow this flow and minimizes ; Bartlett, Long and Bousquet (2023) describe the same dynamics as "bouncing across ravines and drifting towards wide minima". That is the picture in the figure.
Worked example: GD vs SAM gradient near the floor
In the figure's surface the gradient is . Take the point , almost on the floor, with .
| quantity | GD at w | SAM at w + ε̂ |
|---|---|---|
| evaluation point | (1, 0.01) | (1.0020, 0.3100) |
| loss | 0.1445 | |
| gradient u-component (along the floor) | 0.0002 | 0.1926 |
| gradient v-component (across the floor) | 0.0300 | 0.9325 |
- 1, , so : almost straight across the valley.
- 2At , . To first order this is .
- 3The -component overshoots the floor (from to at ), so the next points the other way: SAM zigzags across the floor while the -component, which has the same sign on both sides, accumulates.
- 4At the start , far from the floor: , , and vs . Here SAM behaves like GD with a somewhat larger gradient.
1import torch23def sam_step(model, loss_fn, x, y, base_opt, rho=0.05, adaptive=False):4 """One SAM (or ASAM if adaptive=True) update. Returns the loss at w."""5 params = [p for p in model.parameters() if p.requires_grad]67 # pass 1: gradient at w8 loss = loss_fn(model(x), y)9 loss.backward()1011 with torch.no_grad():12 # ε̂ = ρ g/‖g‖ (SAM) or ρ T²g/‖Tg‖ with T = |w| (ASAM); norm over ALL parameters13 scaled = [(p.abs() * p.grad) if adaptive else p.grad for p in params if p.grad is not None]14 norm = torch.norm(torch.stack([s.norm(2) for s in scaled]), 2)15 eps = []16 for p in params:17 if p.grad is None:18 eps.append(None)19 continue20 e = (p.pow(2) if adaptive else 1.0) * p.grad * (rho / (norm + 1e-12))21 p.add_(e) # climb to w + ε̂22 eps.append(e)23 base_opt.zero_grad()2425 # pass 2: gradient at w + ε̂, same batch26 loss_fn(model(x), y).backward()2728 with torch.no_grad():29 for p, e in zip(params, eps):30 if e is not None:31 p.sub_(e) # back to w32 base_opt.step() # step from w with the SAM gradient33 base_opt.zero_grad()34 return loss.detach()3536model = torch.nn.Sequential(torch.nn.Linear(20, 64), torch.nn.ReLU(), torch.nn.Linear(64, 10))37base = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9, weight_decay=5e-4)38loss_fn = torch.nn.CrossEntropyLoss()39for x, y in [(torch.randn(128, 20), torch.randint(0, 10, (128,)))] * 100:40 sam_step(model, loss_fn, x, y, base, rho=0.05)Variants: m-sharpness, ASAM and cheaper SAMs
m-sharpness. In data-parallel training each worker can compute its own from its own shard of examples, without synchronizing the first pass. Foret et al. found that this per-shard version generalizes better than computing from the whole batch, and that smaller helped; Andriushchenko and Flammarion (ICML 2022) studied why. So the practical SAM is really "-SAM", and the unsynchronized version is also cheaper.
ASAM (Kwon, Kim, Park and Choi, ICML 2021) answers the reparameterization critique. It measures the neighbourhood relative to each weight's size, with , so rescaling a layer no longer changes the measured sharpness. The perturbation becomes (the adaptive=True branch in the code), and is typically an order of magnitude larger than SAM's.
Cost. Two forward-backward passes per step double the compute. Cheaper variants reuse the perturbation for several steps (LookSAM), perturb only a subset of weights or examples (ESAM), or schedule ; GSAM adds a surrogate gap term. In practice people also compare SAM against plain training run for twice as many epochs, the fair compute-matched baseline.
Use when
- +Consistent generalization gains on image classification, especially for ViTs and MLP-Mixers trained from scratch
- +More robust to label noise
- +Wraps any base optimizer; one extra hyperparameter, ρ
- +Explicit, tunable bias toward low-curvature solutions
Avoid when
- −Two forward-backward passes: about twice the compute per step
- −Sharpness is not reparameterization-invariant (motivates ASAM)
- −Extra care with BatchNorm statistics, mixed precision and distributed gradients
- −Gains are smaller or absent on some tasks, and should be compared against training 2× longer
Check yourself
In SAM, at which weights is the gradient that the base optimizer finally uses evaluated, and where is the update applied?
Check yourself
Why is the SAM perturbation ε̂ = ρ g/‖g‖?
Remember
- ✓SAM minimizes : training loss plus a sharpness penalty.
- ✓First-order solution: ; use as the gradient at , dropping the Hessian term.
- ✓Two forward-backward passes per step on the same batch; wrap any base optimizer. PyTorch has no built-in SAM.
- ✓Near a valley of minima SAM bounces across and drifts along the floor toward lower , the flat minima that tend to generalize better.
- ✓Variants: m-sharpness (per-shard , often better), ASAM (scale-invariant, ), and cheaper LookSAM/ESAM; typical .