# Sharpness-aware minimization (SAM)

> Minimize the worst loss in a ρ-ball around the weights, not the loss at a point. One extra gradient step to the approximate worst case, ε ≈ ρ g/‖g‖, steers training toward flat minima that generalize better, for about twice the compute.

Source: https://www.mlfoundry.in/learn/sharpness-aware-minimization-sam  
Section: Deep Learning › Optimization Algorithms  
Level: advanced · 12 min read · updated 2026-10-11

**Sharpness-aware minimization:** (SAM; Foret, Kleiner, Mobahi and Neyshabur, ICLR 2021) trains on the min-max objective $\min_w\max_{\|\epsilon\|_2\le\rho}L(w+\epsilon)$: the loss at the **worst** point within distance $\rho$ of the weights. Each step first moves the weights uphill to $w+\hat\epsilon$, $\hat\epsilon=\rho\,g/\|g\|$, takes the gradient there, and then applies that gradient at the original $w$.

Over-parameterized networks have many weight settings with near-zero training loss, and they do not all generalize equally well. A long-standing observation is that **flat** minima, where the loss stays low in a whole neighbourhood, tend to generalize better than **sharp** ones, where a small change of the weights raises the loss steeply. A rough intuition: the test loss surface is a slightly shifted copy of the training surface, and a shift costs little in a wide basin and a lot in a narrow one.

Hochreiter and Schmidhuber ("Flat minima", 1997) proposed searching for flat regions explicitly. Keskar et al. (ICLR 2017) linked the generalization gap of large-batch training to convergence to sharp minima. Dinh et al. (ICML 2017) pointed out a caveat: for ReLU networks you can rescale weights to make any minimum look arbitrarily sharp without changing the function, so sharpness must be measured carefully. Jiang et al. (ICLR 2020), comparing dozens of generalization measures, still found sharpness-based ones among the most predictive. SAM turned this idea into a simple, scalable optimizer. It improved results on CIFAR-10/100 and ImageNet, made training robust to label noise, and let vision transformers and MLP-Mixers trained from scratch on ImageNet match or beat ResNets of similar size (Chen, Hsieh and Gong, ICLR 2022).

## The objective and its approximation

$$
\min_w\; L^{\text{SAM}}(w)+\lambda\|w\|_2^2,\qquad L^{\text{SAM}}(w)=\max_{\|\epsilon\|_2\le\rho}L(w+\epsilon)=L(w)+\underbrace{\Big[\max_{\|\epsilon\|_2\le\rho}L(w+\epsilon)-L(w)\Big]}_{\text{sharpness}}
$$

- $\rho$: neighbourhood radius; Foret et al. use $\rho=0.05$ for most CIFAR and ImageNet runs
- $\epsilon$: an adversarial weight perturbation, not an input perturbation
- $\lambda\|w\|^2$: ordinary weight decay, kept as usual

The inner maximization is intractable exactly, so SAM linearizes: $L(w+\epsilon)\approx L(w)+\epsilon^\top g$ with $g=\nabla L(w)$. Maximizing a linear function over a ball puts $\epsilon$ on the boundary, in the direction of the gradient:

$$
\hat\epsilon(w)=\arg\max_{\|\epsilon\|_2\le\rho}\epsilon^\top g=\rho\,\frac{g}{\|g\|_2},\qquad \nabla_w L^{\text{SAM}}(w)\approx\nabla L(w)\big|_{w+\hat\epsilon(w)}
$$

- $\hat\epsilon$: one normalized gradient-ascent step of length $\rho$: the worst direction to first order. For a general $\ell_p$ ball it is $\rho\,\operatorname{sign}(g)|g|^{q-1}/\|g\|_q^{q-1}$ with $1/p+1/q=1$
- $\nabla L|_{w+\hat\epsilon}$: the gradient **evaluated at** the perturbed point. The exact gradient also has a term through $d\hat\epsilon/dw$, which involves the Hessian; SAM drops it, and Foret et al. found that keeping it made results worse

**One SAM step (two forward-backward passes)**

1. Pass 1: compute $g=\nabla L_B(w)$ on the mini-batch $B$.
2. Perturb: $\hat\epsilon=\rho\,g/\|g\|$, with the norm taken over **all** parameters together, and move to $w+\hat\epsilon$.
3. Pass 2: compute $g_{\text{SAM}}=\nabla L_B(w+\hat\epsilon)$ on the **same** batch.
4. Restore $w$ (subtract $\hat\epsilon$) and give $g_{\text{SAM}}$ to the base optimizer (SGD with momentum, AdamW, …): $w\leftarrow\text{BaseStep}(w,\,g_{\text{SAM}})$.

> Figure: Every point on the floor v = 0 of L = ½(1 + 2u²)v² is a global minimum with zero training loss, but the curvature across the floor, λmax = 1 + 2u², is 1 in the middle and grows toward the ends. The test loss is the same surface shifted by 0.3, so at a minimum it equals 0.045·λmax. From the start (2.2, 1.5), GD at lr 0.05 drops straight onto the floor and stops at u = 1.556, λmax = 5.84, test loss 0.263. SAM with ρ = 0.3 reaches the floor too, then bounces across it and drifts along it: after 200 steps it is at u = 0.187, λmax = 1.07, test loss 0.051, with a training loss of 3.6e-5 (not 0, because it keeps bouncing) and twice as many gradient evaluations (400). The ring is the ρ-ball, the violet segment is ε̂, and the orange arrow is −∇L(w + ε̂), the direction SAM actually steps in. ρ = 0.1 stops at λmax 3.68; ρ = 0.5 reaches 1.00.

## Why SAM drifts toward flat regions

Near a minimum the gradient is small, so the gradient-ascent step $\hat\epsilon$ points across the valley, along the sharpest direction, and $L(w+\hat\epsilon)\approx L(w)+\tfrac12\rho^2\lambda_{\max}$. Its gradient therefore contains a term $\tfrac12\rho^2\nabla\lambda_{\max}$, which pushes along the valley floor toward lower curvature, even though the training loss itself is already flat in that direction. Wen, Ma and Li ("How does sharpness-aware minimization minimize sharpness?", 2022) proved that full-batch SAM near a manifold of minima does follow this flow and minimizes $\lambda_{\max}$; Bartlett, Long and Bousquet (2023) describe the same dynamics as "bouncing across ravines and drifting towards wide minima". That is the picture in the figure.

## Worked example: GD vs SAM gradient near the floor

In the figure's surface $L(u,v)=\tfrac12(1+2u^2)v^2$ the gradient is $\nabla L=(2uv^2,\;(1+2u^2)v)$. Take the point $(u,v)=(1,\,0.01)$, almost on the floor, with $\rho=0.3$.

Computed with the figure's code. GD barely moves along the floor; SAM's u-gradient is about 960 times larger.

| quantity | GD at w | SAM at w + ε̂ |
| --- | --- | --- |
| evaluation point | (1, 0.01) | (1.0020, 0.3100) |
| loss | $1.5\times10^{-4}$ | 0.1445 |
| gradient u-component (along the floor) | 0.0002 | 0.1926 |
| gradient v-component (across the floor) | 0.0300 | 0.9325 |

1. $g=(0.0002,\;0.03)$, $\|g\|=0.03$, so $\hat\epsilon=0.3\,g/\|g\|=(0.0020,\;0.2999)$: almost straight across the valley.
2. At $w+\hat\epsilon=(1.002,\,0.310)$, $\partial L/\partial u=2\cdot1.002\cdot0.310^2=0.1926$. To first order this is $\tfrac12\rho^2\,\partial\lambda_{\max}/\partial u=\tfrac12\cdot0.09\cdot4u=0.18$.
3. The $v$-component $0.93$ overshoots the floor (from $v=0.01$ to $v\approx-0.036$ at $\eta=0.05$), so the next $\hat\epsilon$ points the other way: SAM zigzags across the floor while the $u$-component, which has the same sign on both sides, accumulates.
4. At the start $(2.2,\,1.5)$, far from the floor: $L=12.0$, $\hat\epsilon=(0.158,\,0.255)$, $L(w+\hat\epsilon)=18.7$ and $g_{\text{SAM}}=(14.5,\,21.3)$ vs $g=(9.9,\,16.0)$. Here SAM behaves like GD with a somewhat larger gradient.

```python
import torch

def sam_step(model, loss_fn, x, y, base_opt, rho=0.05, adaptive=False):
    """One SAM (or ASAM if adaptive=True) update. Returns the loss at w."""
    params = [p for p in model.parameters() if p.requires_grad]

    # pass 1: gradient at w
    loss = loss_fn(model(x), y)
    loss.backward()

    with torch.no_grad():
        # ε̂ = ρ g/‖g‖ (SAM) or ρ T²g/‖Tg‖ with T = |w| (ASAM); norm over ALL parameters
        scaled = [(p.abs() * p.grad) if adaptive else p.grad for p in params if p.grad is not None]
        norm = torch.norm(torch.stack([s.norm(2) for s in scaled]), 2)
        eps = []
        for p in params:
            if p.grad is None:
                eps.append(None)
                continue
            e = (p.pow(2) if adaptive else 1.0) * p.grad * (rho / (norm + 1e-12))
            p.add_(e)                          # climb to w + ε̂
            eps.append(e)
    base_opt.zero_grad()

    # pass 2: gradient at w + ε̂, same batch
    loss_fn(model(x), y).backward()

    with torch.no_grad():
        for p, e in zip(params, eps):
            if e is not None:
                p.sub_(e)                      # back to w
    base_opt.step()                            # step from w with the SAM gradient
    base_opt.zero_grad()
    return loss.detach()

model = torch.nn.Sequential(torch.nn.Linear(20, 64), torch.nn.ReLU(), torch.nn.Linear(64, 10))
base = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9, weight_decay=5e-4)
loss_fn = torch.nn.CrossEntropyLoss()
for x, y in [(torch.randn(128, 20), torch.randint(0, 10, (128,)))] * 100:
    sam_step(model, loss_fn, x, y, base, rho=0.05)
```

PyTorch has no built-in SAM optimizer: wrap any base optimizer with two forward-backward passes

> **BatchNorm, mixed precision and gradient clipping:** The second pass runs BatchNorm in training mode again and updates its running statistics with activations from the perturbed weights. Common implementations disable the running-stat update for the second pass (for example by setting each BatchNorm's `momentum` to 0 temporarily). With `torch.amp.GradScaler`, unscale the gradients before computing $\|g\|$, otherwise $\hat\epsilon$ is computed from scaled gradients. With data parallelism, decide deliberately whether to all-reduce the first-pass gradient (see m-sharpness below).

## Variants: m-sharpness, ASAM and cheaper SAMs

**m-sharpness.** In data-parallel training each worker can compute its own $\hat\epsilon$ from its own shard of $m$ examples, without synchronizing the first pass. Foret et al. found that this per-shard version generalizes **better** than computing $\hat\epsilon$ from the whole batch, and that smaller $m$ helped; Andriushchenko and Flammarion (ICML 2022) studied why. So the practical SAM is really "$m$-SAM", and the unsynchronized version is also cheaper.

**ASAM** (Kwon, Kim, Park and Choi, ICML 2021) answers the reparameterization critique. It measures the neighbourhood relative to each weight's size, $\|T_w^{-1}\epsilon\|\le\rho$ with $T_w=\operatorname{diag}(|w|)$, so rescaling a layer no longer changes the measured sharpness. The perturbation becomes $\hat\epsilon=\rho\,T_w^2g/\|T_wg\|$ (the `adaptive=True` branch in the code), and $\rho$ is typically an order of magnitude larger than SAM's.

**Cost.** Two forward-backward passes per step double the compute. Cheaper variants reuse the perturbation for several steps (LookSAM), perturb only a subset of weights or examples (ESAM), or schedule $\rho$; GSAM adds a surrogate gap term. In practice people also compare SAM against plain training run for **twice as many epochs**, the fair compute-matched baseline.

> **Large learning rates and small batches already do some of this:** Gradient descent cannot settle where $\lambda_{\max}>2/\eta$: such minima are unstable for it. A large learning rate therefore implicitly avoids sharp minima, and SGD noise does too. In the figure, GD alone at $\eta=0.15$ lands at $\lambda_{\max}=1.31$ instead of 5.84 at $\eta=0.05$, because its larger steps bounce across the valley for longer, and that bouncing, like SAM's, pushes it along the floor. SAM makes the preference explicit and controllable through $\rho$, independently of the learning rate.

> **Choosing ρ:** $\rho$ is in parameter units, so it interacts with weight scale, normalization and weight decay. Foret et al. tuned it over $\{0.01,0.02,0.05,0.1,0.2,0.5\}$; 0.05 is the usual first try for SAM with SGD on vision models. Too large a $\rho$ evaluates the gradient where the linear approximation no longer holds and slows or destabilizes training; $\rho=0$ recovers the base optimizer exactly.

**Strengths**

- Consistent generalization gains on image classification, especially for ViTs and MLP-Mixers trained from scratch
- More robust to label noise
- Wraps any base optimizer; one extra hyperparameter, ρ
- Explicit, tunable bias toward low-curvature solutions

**Limitations**

- Two forward-backward passes: about twice the compute per step
- Sharpness is not reparameterization-invariant (motivates ASAM)
- Extra care with BatchNorm statistics, mixed precision and distributed gradients
- Gains are smaller or absent on some tasks, and should be compared against training 2× longer

**Question:** In SAM, at which weights is the gradient that the base optimizer finally uses evaluated, and where is the update applied?

- A. Evaluated at w + ε̂, applied starting from w
- B. Evaluated at w, applied starting from w + ε̂
- C. Evaluated at w − ε̂, applied starting from w
- D. Evaluated at w, and ε̂ is simply added to the update

**Answer:** A. Evaluated at w + ε̂, applied starting from w — SAM climbs to $w+\hat\epsilon$, takes $g_{\text{SAM}}=\nabla L(w+\hat\epsilon)$ there, then subtracts $\hat\epsilon$ again and lets the base optimizer step from $w$ with $g_{\text{SAM}}$. The perturbation is only a probe for the worst nearby gradient.

**Question:** Why is the SAM perturbation ε̂ = ρ g/‖g‖?

- A. It is the exact solution of the inner maximization for any loss
- B. It maximizes the first-order approximation L(w) + εᵀg over the ball ‖ε‖ ≤ ρ
- C. It minimizes the loss inside the ball
- D. It is chosen to make the Hessian diagonal

**Answer:** B. It maximizes the first-order approximation L(w) + εᵀg over the ball ‖ε‖ ≤ ρ — A linear function $\epsilon^\top g$ is maximized over a Euclidean ball at the boundary point in the direction of $g$, which gives $\rho\,g/\|g\|$. It is exact only to first order; that is why SAM is one step of normalized gradient ascent, not a full inner optimization.

**Key points**

- SAM minimizes $\max_{\|\epsilon\|\le\rho}L(w+\epsilon)$: training loss plus a sharpness penalty.
- First-order solution: $\hat\epsilon=\rho\,g/\|g\|$; use $\nabla L(w+\hat\epsilon)$ as the gradient at $w$, dropping the Hessian term.
- Two forward-backward passes per step on the same batch; wrap any base optimizer. PyTorch has no built-in SAM.
- Near a valley of minima SAM bounces across and drifts along the floor toward lower $\lambda_{\max}$, the flat minima that tend to generalize better.
- Variants: m-sharpness (per-shard $\hat\epsilon$, often better), ASAM (scale-invariant, $T_w=|w|$), and cheaper LookSAM/ESAM; typical $\rho=0.05$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
