# Gradient noise

> Add Gaussian noise to every gradient, with variance annealed as η/(1 + t)^γ. It is Langevin dynamics at a falling temperature: it escapes saddles and shallow basins early and settles later. SGD already has its own noise, and the evidence for adding more is mixed.

Source: https://www.mlfoundry.in/learn/gradient-noise  
Section: Deep Learning › Regularization & Normalization  
Level: advanced · 11 min read · updated 2026-10-11

**Gradient noise:** (annealed Gaussian gradient noise) adds a sample from $\mathcal N(0,\sigma_t^2 I)$ to the gradient before each optimizer step, with a variance $\sigma_t^2$ that decays over training. Early on the noise is large and the parameters wander; later it fades and the optimizer converges as usual.

Deliberately corrupting the gradient sounds counterproductive, but noise has a long history in optimization. Simulated annealing (Kirkpatrick, Gelatt and Vecchi, 1983) accepts uphill moves with a probability that shrinks as a temperature is lowered. Gelfand and Mitter (1991) proved that gradient descent plus Gaussian noise whose variance decays slowly enough (like $1/\log t$) converges to global minimizers. In deep learning the question is more practical: can a little extra noise help a hard-to-train network find better regions early in training?

Neelakantan, Vilnis, Le, Sutskever, Kaiser, Kurach and Martens (2015), "Adding gradient noise improves learning for very deep networks", proposed the simple schedule below and reported gains on hard optimization problems: a 20-layer fully connected network on MNIST with poor initialization, and complex models such as End-to-End Memory Networks, the Neural Programmer and Neural Random Access Machines. Separately, Welling and Teh (2011) had shown that adding properly scaled noise to SGD turns it into a **sampler** of the Bayesian posterior: stochastic gradient Langevin dynamics (SGLD).

## The update

$$
\theta_{t+1}=\theta_t-\alpha\big(g_t+\epsilon_t\big),\qquad \epsilon_t\sim\mathcal N\!\big(0,\ \sigma_t^2 I\big),\qquad \sigma_t^2=\frac{\eta}{(1+t)^{\gamma}}
$$

- $g_t$: the (mini-batch) gradient $\nabla_\theta L$ at step $t$; with Adam etc. the noisy gradient is fed to the optimizer instead
- $\alpha$: the learning rate
- $\eta$: the noise scale (not the learning rate here, following the paper's notation); they tried $\eta\in\{0.01,\,0.3,\,1.0\}$
- $\gamma$: the decay exponent, $\gamma=0.55$ in the paper; $\gamma=0$ is constant noise

The noise is independent of the data, isotropic (the same variance in every direction) and has zero mean, so the expected update is unchanged. What changes is the spread of the trajectory. Per step the parameters receive an extra random displacement of variance $\alpha^2\sigma_t^2$ in each coordinate.

## Langevin dynamics: noise as temperature

The continuous-time **Langevin equation** $d\theta=-\nabla f(\theta)\,dt+\sqrt{2T}\,dW_t$ has a known long-run behaviour: whatever the start, $\theta$ ends up distributed according to the Gibbs density $p(\theta)\propto e^{-f(\theta)/T}$. Its Euler discretization with step $\alpha$ is

$$
\theta_{t+1}=\theta_t-\alpha\,\nabla f(\theta_t)+\sqrt{2\alpha T}\;\xi_t,\quad \xi_t\sim\mathcal N(0,I)\qquad\Longleftrightarrow\qquad T=\frac{\alpha\,\sigma^2}{2}
$$

- $T$: temperature: at $T\to0$ the density concentrates on the global minima; at large $T$ it spreads over every basin
- $T=\alpha\sigma^2/2$: matching the per-step variances $\alpha^2\sigma^2=2\alpha T$: gradient noise of variance $\sigma^2$ at learning rate $\alpha$ is Langevin dynamics at temperature $\alpha\sigma^2/2$

So annealed gradient noise is Langevin dynamics with a **decreasing temperature** $T_t=\alpha\eta/(2(1+t)^\gamma)$: a continuous form of simulated annealing. Early, the temperature is high and the iterate can climb over barriers. Escaping a basin of depth $\Delta f$ takes a time roughly proportional to $e^{\Delta f/T}$ (Arrhenius–Kramers), so the deepest, widest basins are the ones it settles in as $T$ falls. The polynomial schedule cools far faster than the $1/\log t$ that the global-convergence theory requires, so there is no guarantee of finding the global minimum. It is a heuristic for exploring early, not a global optimizer.

**SGLD** (Welling and Teh, 2011) uses the same update with $T=1$ and $f$ the negative log-posterior, $f(\theta)=-\ln p(\theta)-\tfrac Nn\sum_{i\in\mathcal B}\ln p(x_i\mid\theta)$, and a step size that decreases to zero. As the step shrinks, the injected noise $\sqrt{2\alpha}$ dominates the mini-batch noise (which scales like $\alpha$), and the iterates become approximate samples from the posterior. Averaging predictions over those samples gives a cheap Bayesian ensemble.

**At a glance**

- **Hyperparameters:** noise scale $\eta$ and decay $\gamma$ (paper: $\eta\in\{0.01,0.3,1\}$, $\gamma=0.55$)
- **Cost:** one Gaussian sample per parameter per step
- **Mechanism:** Langevin dynamics at temperature $T_t=\alpha\sigma_t^2/2$, cooled over training
- **Helps with:** saddles and flat regions, poor initialization, hard-to-optimize architectures
- **Interacts with:** SGD's own mini-batch noise, adaptive optimizers (which rescale the noise), gradient clipping
- **PyTorch:** no built-in: add `torch.randn_like(p.grad)` to each gradient before `opt.step()`

> Figure: 200 particles on the tilted double well f(x) = x⁴/4 − x² + 0.3x, with learning rate α = 0.05: shallow minimum at x = 1.33 (f = −0.588), deep minimum at −1.48 (f = −1.435), barrier top at 0.15. Plain GD started in the shallow well stays there: 0% reach the deep basin. Annealed noise with η = 200, γ = 0.55 starts at temperature ασ²/2 = 5.0 and cools to 0.112 by step 1000: 17.5% of the particles have crossed by step 5, 52.5% by step 100 and 76.5% by step 1000, with mean f = −1.171. Set γ = 0 (constant noise): about 60% are in the deep basin but they never settle (mean f = +0.081). Set γ = 1: it cools too fast and only 19% cross. Langevin dynamics at T = 0.3 approaches the Gibbs density exp(−f/T)/Z: 85.0% in the deep basin at step 1000, against 93.1% at equilibrium; at T = 0.1 only 4.5% have crossed. Start at the stationary point instead: GD never moves (f′ = 0), while any noise sends about half the particles each way within 5 steps.

## Worked example: schedule and barrier

The paper's schedule with η = 0.3, γ = 0.55 (computed in Node)

| step t | σₜ² = 0.3/(1 + t)^0.55 | σₜ | relative to t = 0 |
| --- | --- | --- | --- |
| 0 | 0.300 | 0.548 | 1 |
| 10 | 0.0802 | 0.283 | 0.52 |
| 100 | 0.0237 | 0.154 | 0.28 |
| 1 000 | 0.00671 | 0.0819 | 0.15 |
| 10 000 | 0.00189 | 0.0435 | 0.079 |

With $\gamma=0.55$ the standard deviation falls like $t^{-0.275}$: it halves in the first 10 steps but then decays slowly, so a long run keeps a non-trivial amount of noise for thousands of steps. The noise is in absolute units, so whether $\sigma=0.08$ is large depends on the typical gradient norm of the model. That is one reason the useful $\eta$ varies across tasks.

**Why temperature matters so much: the figure's barrier**

1. Barrier height from the shallow minimum: $f(0.152)-f(1.332)=0.0226-(-0.5877)=0.610$.
2. Arrhenius factor $e^{\Delta f/T}$: $e^{0.610/0.3}=7.6$ at $T=0.3$, but $e^{0.610/0.1}=447$ at $T=0.1$. Cooling by a factor of 3 makes escape about 59 times slower.
3. That matches the figure: at $T=0.3$, 85% of the particles have crossed by step 1000; at $T=0.1$, only 4.5%.
4. Going back out of the deep well needs $\Delta f=0.0226-(-1.435)=1.458$, so $e^{1.458/0.3}\approx129$: 17 times harder than leaving the shallow well at the same temperature. This asymmetry is why most particles end in the deeper basin.

```python
import torch

def add_gradient_noise(params, step, eta=0.3, gamma=0.55):
    """g <- g + N(0, eta / (1 + step)^gamma), in place, before opt.step()."""
    std = (eta / (1 + step) ** gamma) ** 0.5
    for p in params:
        if p.grad is not None:
            p.grad.add_(torch.randn_like(p.grad), alpha=std)

# for step, (x, y) in enumerate(loader):
#     opt.zero_grad(set_to_none=True)
#     loss_fn(model(x), y).backward()
#     add_gradient_noise(model.parameters(), step)
#     opt.step()

@torch.no_grad()
def sgld_step(params, lr, temperature=1.0):
    """theta <- theta - lr * grad + sqrt(2 * lr * T) * xi.
    For posterior sampling the loss must be the full negative log-posterior:
    N * mean_batch_NLL + (-log prior), not the mean loss."""
    for p in params:
        if p.grad is None:
            continue
        p.add_(p.grad, alpha=-lr)
        p.add_(torch.randn_like(p), alpha=(2 * lr * temperature) ** 0.5)
```

Annealed gradient noise (Neelakantan et al.) and an SGLD step, in plain PyTorch

## SGD already has noise

A mini-batch gradient is the full gradient plus noise with covariance $\Sigma(\theta)/B$. Plain SGD therefore already behaves like Langevin dynamics at an effective temperature of order $\alpha\,\operatorname{tr}\Sigma/B$ (Mandt, Hoffman and Blei, 2017; see **Stochastic gradient descent (SGD)** and **Mini-batch gradient descent**, where the noise scale $\alpha/B$ appears). Two differences matter. SGD's noise is **anisotropic**: near a minimum its covariance is aligned with the curvature, large along sharp directions and small along flat ones. And it vanishes on examples the model already fits. Isotropic Gaussian noise of the same size spends most of its variance in directions that do not matter.

Zhu, Wu, Yu, Wu and Ma (2019) showed that this structure is what lets SGD escape sharp minima: in their experiments, isotropic noise of comparable magnitude did not reproduce SGD's regularization effect, while noise with the gradient covariance's structure did. This also explains why adding Gaussian noise to large-batch training is not a reliable substitute for a small batch.

## Escaping saddles

In high dimensions most stationary points of a non-convex loss are saddles, not minima. Exact gradient descent started exactly on a saddle's stable manifold never leaves (the figure's stationary-point start shows the one-dimensional version). From a random initialization GD almost surely avoids strict saddles (Lee et al., 2016), but it can take exponentially long to get past a sequence of them (Du et al., 2017). Noise fixes this: Ge, Huang, Jin and Yuan (2015) showed that noisy SGD escapes strict saddles in polynomial time, and Jin et al. (2017) showed that a single small isotropic perturbation, added whenever the gradient is small, escapes with only a polylogarithmic dependence on the dimension. This is the cleanest theoretical case for gradient noise.

> **Mixed evidence in practice:** The original gains were largest on hard optimization problems: very deep networks with poor initialization and complex memory- and program-learning architectures. With modern defaults (He initialization, residual connections, normalization layers, Adam or SGD with momentum and a tuned schedule) many reports find little or no benefit, and a badly chosen $\eta$ simply slows convergence. Treat gradient noise as a tool for networks that will not train, and verify the gain against a run without it.

> **Scale and optimizer interactions:** $\sigma_t$ is in absolute gradient units, so the right $\eta$ depends on the loss scale and the model. With Adam, the noise also enters the second-moment estimate and is then divided by $\sqrt{\hat v}$, so its effective size is not what the schedule says. If you clip gradients, decide whether noise goes before clipping (it gets clipped too) or after (it can exceed the clip).

> **Explore early, exploit late:** The annealing schedule mirrors the learning-rate schedule. Large noise or a large step early lets the iterate move between basins. Small noise and a small step late let it settle in one. A warm-up followed by a decaying learning rate already supplies much of this effect through SGD's own noise, which is part of why explicit gradient noise rarely helps well-tuned modern training.

> **Sampling, not just optimizing:** With $T=1$, a decreasing step and the full negative log-posterior as the loss, the same update is SGLD: iterates after burn-in are approximate posterior samples, and averaging their predictions gives uncertainty estimates. Cyclical step sizes (Zhang et al., 2020, cSG-MCMC) help it visit several modes in one run.

**Strengths**

- Two lines of code; no extra forward or backward passes
- Escapes saddles and flat regions that exact GD can stall on
- Annealing gives early exploration and late convergence
- With the Langevin scaling, the same update samples a posterior (SGLD)

**Limitations**

- Isotropic noise is a poor match for the loss geometry; SGD's own anisotropic noise is usually more useful
- Little or no benefit on well-initialized, normalized, residual networks
- Absolute-scale hyperparameter η that must be tuned per model; interacts with Adam and clipping
- Polynomial cooling gives no global-optimum guarantee

**Question:** You train with learning rate α = 0.01 and add gradient noise of standard deviation σ = 2. In the Langevin picture, what temperature does this correspond to?

- A. 0.02
- B. 0.04
- C. 0.01
- D. 2

**Answer:** A. 0.02 — Matching the per-step variance $\alpha^2\sigma^2=2\alpha T$ gives $T=\alpha\sigma^2/2=0.01\cdot4/2=0.02$. Halving the learning rate at fixed σ halves the temperature, which is why noise scale and learning rate must be tuned together.

**Question:** In the figure, constant noise (γ = 0) puts about 60% of the particles in the deep basin, yet their mean loss is worse than with γ = 0.55. Why?

- A. Constant noise has zero mean
- B. With constant noise the temperature never falls, so the particles keep bouncing around at a high loss instead of settling into the minimum
- C. γ = 0 makes the noise vanish
- D. The deep basin is unstable

**Answer:** B. With constant noise the temperature never falls, so the particles keep bouncing around at a high loss instead of settling into the minimum — At a fixed temperature the particles sample a spread-out Gibbs density, which at T = 5 covers much of the landscape, so the mean f stays high (+0.081). Annealing lowers T so that, once the particles have found the deep basin, they also settle at its bottom (mean f −1.171).

**Key points**

- Update $\theta\leftarrow\theta-\alpha(g_t+\epsilon_t)$ with $\epsilon_t\sim\mathcal N(0,\sigma_t^2I)$ and $\sigma_t^2=\eta/(1+t)^\gamma$ (Neelakantan et al., 2015: $\gamma=0.55$).
- It is Langevin dynamics at temperature $T_t=\alpha\sigma_t^2/2$: a cooling schedule that explores early and settles late; escape time grows like $e^{\Delta f/T}$.
- With T = 1, a decreasing step and the log-posterior as loss, the same update is SGLD, a posterior sampler (Welling and Teh, 2011).
- SGD already injects anisotropic, curvature-aligned noise of scale ~α/B; isotropic Gaussian noise is usually a weaker regularizer than that.
- Clear theoretical benefit for escaping saddles; mixed empirical evidence on modern, well-tuned networks.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
