Definition
Gradient noise (annealed Gaussian gradient noise) adds a sample from to the gradient before each optimizer step, with a variance that decays over training. Early on the noise is large and the parameters wander; later it fades and the optimizer converges as usual.
Deliberately corrupting the gradient sounds counterproductive, but noise has a long history in optimization. Simulated annealing (Kirkpatrick, Gelatt and Vecchi, 1983) accepts uphill moves with a probability that shrinks as a temperature is lowered. Gelfand and Mitter (1991) proved that gradient descent plus Gaussian noise whose variance decays slowly enough (like ) converges to global minimizers. In deep learning the question is more practical: can a little extra noise help a hard-to-train network find better regions early in training?
Neelakantan, Vilnis, Le, Sutskever, Kaiser, Kurach and Martens (2015), "Adding gradient noise improves learning for very deep networks", proposed the simple schedule below and reported gains on hard optimization problems: a 20-layer fully connected network on MNIST with poor initialization, and complex models such as End-to-End Memory Networks, the Neural Programmer and Neural Random Access Machines. Separately, Welling and Teh (2011) had shown that adding properly scaled noise to SGD turns it into a sampler of the Bayesian posterior: stochastic gradient Langevin dynamics (SGLD).
The update
- the (mini-batch) gradient at step ; with Adam etc. the noisy gradient is fed to the optimizer instead
- the learning rate
- the noise scale (not the learning rate here, following the paper's notation); they tried
- the decay exponent, in the paper; is constant noise
The noise is independent of the data, isotropic (the same variance in every direction) and has zero mean, so the expected update is unchanged. What changes is the spread of the trajectory. Per step the parameters receive an extra random displacement of variance in each coordinate.
Langevin dynamics: noise as temperature
The continuous-time Langevin equation has a known long-run behaviour: whatever the start, ends up distributed according to the Gibbs density . Its Euler discretization with step is
- temperature: at the density concentrates on the global minima; at large it spreads over every basin
- matching the per-step variances : gradient noise of variance at learning rate is Langevin dynamics at temperature
So annealed gradient noise is Langevin dynamics with a decreasing temperature : a continuous form of simulated annealing. Early, the temperature is high and the iterate can climb over barriers. Escaping a basin of depth takes a time roughly proportional to (Arrhenius–Kramers), so the deepest, widest basins are the ones it settles in as falls. The polynomial schedule cools far faster than the that the global-convergence theory requires, so there is no guarantee of finding the global minimum. It is a heuristic for exploring early, not a global optimizer.
SGLD (Welling and Teh, 2011) uses the same update with and the negative log-posterior, , and a step size that decreases to zero. As the step shrinks, the injected noise dominates the mini-batch noise (which scales like ), and the iterates become approximate samples from the posterior. Averaging predictions over those samples gives a cheap Bayesian ensemble.
At a glance
- Hyperparameters
- noise scale and decay (paper: , )
- Cost
- one Gaussian sample per parameter per step
- Mechanism
- Langevin dynamics at temperature , cooled over training
- Helps with
- saddles and flat regions, poor initialization, hard-to-optimize architectures
- Interacts with
- SGD's own mini-batch noise, adaptive optimizers (which rescale the noise), gradient clipping
- PyTorch
- no built-in: add
torch.randn_like(p.grad)to each gradient beforeopt.step()
Worked example: schedule and barrier
| step t | σₜ² = 0.3/(1 + t)^0.55 | σₜ | relative to t = 0 |
|---|---|---|---|
| 0 | 0.300 | 0.548 | 1 |
| 10 | 0.0802 | 0.283 | 0.52 |
| 100 | 0.0237 | 0.154 | 0.28 |
| 1 000 | 0.00671 | 0.0819 | 0.15 |
| 10 000 | 0.00189 | 0.0435 | 0.079 |
With the standard deviation falls like : it halves in the first 10 steps but then decays slowly, so a long run keeps a non-trivial amount of noise for thousands of steps. The noise is in absolute units, so whether is large depends on the typical gradient norm of the model. That is one reason the useful varies across tasks.
Why temperature matters so much: the figure's barrier
- 1Barrier height from the shallow minimum: .
- 2Arrhenius factor : at , but at . Cooling by a factor of 3 makes escape about 59 times slower.
- 3That matches the figure: at , 85% of the particles have crossed by step 1000; at , only 4.5%.
- 4Going back out of the deep well needs , so : 17 times harder than leaving the shallow well at the same temperature. This asymmetry is why most particles end in the deeper basin.
1import torch23def add_gradient_noise(params, step, eta=0.3, gamma=0.55):4 """g <- g + N(0, eta / (1 + step)^gamma), in place, before opt.step()."""5 std = (eta / (1 + step) ** gamma) ** 0.56 for p in params:7 if p.grad is not None:8 p.grad.add_(torch.randn_like(p.grad), alpha=std)910# for step, (x, y) in enumerate(loader):11# opt.zero_grad(set_to_none=True)12# loss_fn(model(x), y).backward()13# add_gradient_noise(model.parameters(), step)14# opt.step()1516@torch.no_grad()17def sgld_step(params, lr, temperature=1.0):18 """theta <- theta - lr * grad + sqrt(2 * lr * T) * xi.19 For posterior sampling the loss must be the full negative log-posterior:20 N * mean_batch_NLL + (-log prior), not the mean loss."""21 for p in params:22 if p.grad is None:23 continue24 p.add_(p.grad, alpha=-lr)25 p.add_(torch.randn_like(p), alpha=(2 * lr * temperature) ** 0.5)SGD already has noise
A mini-batch gradient is the full gradient plus noise with covariance . Plain SGD therefore already behaves like Langevin dynamics at an effective temperature of order (Mandt, Hoffman and Blei, 2017; see Stochastic gradient descent (SGD) and Mini-batch gradient descent, where the noise scale appears). Two differences matter. SGD's noise is anisotropic: near a minimum its covariance is aligned with the curvature, large along sharp directions and small along flat ones. And it vanishes on examples the model already fits. Isotropic Gaussian noise of the same size spends most of its variance in directions that do not matter.
Zhu, Wu, Yu, Wu and Ma (2019) showed that this structure is what lets SGD escape sharp minima: in their experiments, isotropic noise of comparable magnitude did not reproduce SGD's regularization effect, while noise with the gradient covariance's structure did. This also explains why adding Gaussian noise to large-batch training is not a reliable substitute for a small batch.
Escaping saddles
In high dimensions most stationary points of a non-convex loss are saddles, not minima. Exact gradient descent started exactly on a saddle's stable manifold never leaves (the figure's stationary-point start shows the one-dimensional version). From a random initialization GD almost surely avoids strict saddles (Lee et al., 2016), but it can take exponentially long to get past a sequence of them (Du et al., 2017). Noise fixes this: Ge, Huang, Jin and Yuan (2015) showed that noisy SGD escapes strict saddles in polynomial time, and Jin et al. (2017) showed that a single small isotropic perturbation, added whenever the gradient is small, escapes with only a polylogarithmic dependence on the dimension. This is the cleanest theoretical case for gradient noise.
Use when
- +Two lines of code; no extra forward or backward passes
- +Escapes saddles and flat regions that exact GD can stall on
- +Annealing gives early exploration and late convergence
- +With the Langevin scaling, the same update samples a posterior (SGLD)
Avoid when
- −Isotropic noise is a poor match for the loss geometry; SGD's own anisotropic noise is usually more useful
- −Little or no benefit on well-initialized, normalized, residual networks
- −Absolute-scale hyperparameter η that must be tuned per model; interacts with Adam and clipping
- −Polynomial cooling gives no global-optimum guarantee
Check yourself
You train with learning rate α = 0.01 and add gradient noise of standard deviation σ = 2. In the Langevin picture, what temperature does this correspond to?
Check yourself
In the figure, constant noise (γ = 0) puts about 60% of the particles in the deep basin, yet their mean loss is worse than with γ = 0.55. Why?
Remember
- ✓Update with and (Neelakantan et al., 2015: ).
- ✓It is Langevin dynamics at temperature : a cooling schedule that explores early and settles late; escape time grows like .
- ✓With T = 1, a decreasing step and the log-posterior as loss, the same update is SGLD, a posterior sampler (Welling and Teh, 2011).
- ✓SGD already injects anisotropic, curvature-aligned noise of scale ~α/B; isotropic Gaussian noise is usually a weaker regularizer than that.
- ✓Clear theoretical benefit for escaping saddles; mixed empirical evidence on modern, well-tuned networks.