Gradient noise

Add Gaussian noise to every gradient, with variance annealed as η/(1 + t)^γ. It is Langevin dynamics at a falling temperature: it escapes saddles and shallow basins early and settles later. SGD already has its own noise, and the evidence for adding more is mixed.

11 min read1215 wordsadvanced0 viewsupdated 2026-10-11

Definition

Gradient noise (annealed Gaussian gradient noise) adds a sample from N(0,σt2I)\mathcal N(0,\sigma_t^2 I) to the gradient before each optimizer step, with a variance σt2\sigma_t^2 that decays over training. Early on the noise is large and the parameters wander; later it fades and the optimizer converges as usual.

Deliberately corrupting the gradient sounds counterproductive, but noise has a long history in optimization. Simulated annealing (Kirkpatrick, Gelatt and Vecchi, 1983) accepts uphill moves with a probability that shrinks as a temperature is lowered. Gelfand and Mitter (1991) proved that gradient descent plus Gaussian noise whose variance decays slowly enough (like 1/log⁡t1/\log t) converges to global minimizers. In deep learning the question is more practical: can a little extra noise help a hard-to-train network find better regions early in training?

Neelakantan, Vilnis, Le, Sutskever, Kaiser, Kurach and Martens (2015), "Adding gradient noise improves learning for very deep networks", proposed the simple schedule below and reported gains on hard optimization problems: a 20-layer fully connected network on MNIST with poor initialization, and complex models such as End-to-End Memory Networks, the Neural Programmer and Neural Random Access Machines. Separately, Welling and Teh (2011) had shown that adding properly scaled noise to SGD turns it into a sampler of the Bayesian posterior: stochastic gradient Langevin dynamics (SGLD).

The update

θt+1=θt−α(gt+ϵt),ϵt∼N ⁣(0, σt2I),σt2=η(1+t)γ\theta_{t+1}=\theta_t-\alpha\big(g_t+\epsilon_t\big),\qquad \epsilon_t\sim\mathcal N\!\big(0,\ \sigma_t^2 I\big),\qquad \sigma_t^2=\frac{\eta}{(1+t)^{\gamma}}
gtg_t
the (mini-batch) gradient ∇θL\nabla_\theta L at step tt; with Adam etc. the noisy gradient is fed to the optimizer instead
α\alpha
the learning rate
η\eta
the noise scale (not the learning rate here, following the paper's notation); they tried η∈{0.01, 0.3, 1.0}\eta\in\{0.01,\,0.3,\,1.0\}
γ\gamma
the decay exponent, γ=0.55\gamma=0.55 in the paper; γ=0\gamma=0 is constant noise

The noise is independent of the data, isotropic (the same variance in every direction) and has zero mean, so the expected update is unchanged. What changes is the spread of the trajectory. Per step the parameters receive an extra random displacement of variance α2σt2\alpha^2\sigma_t^2 in each coordinate.

Langevin dynamics: noise as temperature

The continuous-time Langevin equation dθ=−∇f(θ) dt+2T dWtd\theta=-\nabla f(\theta)\,dt+\sqrt{2T}\,dW_t has a known long-run behaviour: whatever the start, θ\theta ends up distributed according to the Gibbs density p(θ)∝e−f(θ)/Tp(\theta)\propto e^{-f(\theta)/T}. Its Euler discretization with step α\alpha is

θt+1=θt−α ∇f(θt)+2αT  ξt,ξt∼N(0,I)⟺T=α σ22\theta_{t+1}=\theta_t-\alpha\,\nabla f(\theta_t)+\sqrt{2\alpha T}\;\xi_t,\quad \xi_t\sim\mathcal N(0,I)\qquad\Longleftrightarrow\qquad T=\frac{\alpha\,\sigma^2}{2}
TT
temperature: at T→0T\to0 the density concentrates on the global minima; at large TT it spreads over every basin
T=ασ2/2T=\alpha\sigma^2/2
matching the per-step variances α2σ2=2αT\alpha^2\sigma^2=2\alpha T: gradient noise of variance σ2\sigma^2 at learning rate α\alpha is Langevin dynamics at temperature ασ2/2\alpha\sigma^2/2

So annealed gradient noise is Langevin dynamics with a decreasing temperature Tt=αη/(2(1+t)γ)T_t=\alpha\eta/(2(1+t)^\gamma): a continuous form of simulated annealing. Early, the temperature is high and the iterate can climb over barriers. Escaping a basin of depth Δf\Delta f takes a time roughly proportional to eΔf/Te^{\Delta f/T} (Arrhenius–Kramers), so the deepest, widest basins are the ones it settles in as TT falls. The polynomial schedule cools far faster than the 1/log⁡t1/\log t that the global-convergence theory requires, so there is no guarantee of finding the global minimum. It is a heuristic for exploring early, not a global optimizer.

SGLD (Welling and Teh, 2011) uses the same update with T=1T=1 and ff the negative log-posterior, f(θ)=−ln⁡p(θ)−Nn∑i∈Bln⁡p(xi∣θ)f(\theta)=-\ln p(\theta)-\tfrac Nn\sum_{i\in\mathcal B}\ln p(x_i\mid\theta), and a step size that decreases to zero. As the step shrinks, the injected noise 2α\sqrt{2\alpha} dominates the mini-batch noise (which scales like α\alpha), and the iterates become approximate samples from the posterior. Averaging predictions over those samples gives a cheap Bayesian ensemble.

At a glance

Hyperparameters
noise scale η\eta and decay γ\gamma (paper: η∈{0.01,0.3,1}\eta\in\{0.01,0.3,1\}, γ=0.55\gamma=0.55)
Cost
one Gaussian sample per parameter per step
Mechanism
Langevin dynamics at temperature Tt=ασt2/2T_t=\alpha\sigma_t^2/2, cooled over training
Helps with
saddles and flat regions, poor initialization, hard-to-optimize architectures
Interacts with
SGD's own mini-batch noise, adaptive optimizers (which rescale the noise), gradient clipping
PyTorch
no built-in: add torch.randn_like(p.grad) to each gradient before opt.step()
f(x) = x⁴/4 − x² + 0.3x · 200 particles · α = 0.05deep min -1.48f′ = 0 0.15shallow min 1.33particle density at step 1000step1000noise std σₜ2.115temperature ασ²/20.112in deep basin76.5%mean f(x)-1.171share in deep basin00.511000 stepsx ← x − α (f′(x) + gₜ), gₜ ~ N(0, η/(1+t)^γ), η = 200, γ = 0.55
start atevery 5 steps shown
fig.200 particles on the tilted double well f(x) = x⁴/4 − x² + 0.3x, with learning rate α = 0.05: shallow minimum at x = 1.33 (f = −0.588), deep minimum at −1.48 (f = −1.435), barrier top at 0.15. Plain GD started in the shallow well stays there: 0% reach the deep basin. Annealed noise with η = 200, γ = 0.55 starts at temperature ασ²/2 = 5.0 and cools to 0.112 by step 1000: 17.5% of the particles have crossed by step 5, 52.5% by step 100 and 76.5% by step 1000, with mean f = −1.171. Set γ = 0 (constant noise): about 60% are in the deep basin but they never settle (mean f = +0.081). Set γ = 1: it cools too fast and only 19% cross. Langevin dynamics at T = 0.3 approaches the Gibbs density exp(−f/T)/Z: 85.0% in the deep basin at step 1000, against 93.1% at equilibrium; at T = 0.1 only 4.5% have crossed. Start at the stationary point instead: GD never moves (f′ = 0), while any noise sends about half the particles each way within 5 steps.

Worked example: schedule and barrier

step tσₜ² = 0.3/(1 + t)^0.55σₜrelative to t = 0
00.3000.5481
100.08020.2830.52
1000.02370.1540.28
1 0000.006710.08190.15
10 0000.001890.04350.079
The paper's schedule with η = 0.3, γ = 0.55 (computed in Node)

With γ=0.55\gamma=0.55 the standard deviation falls like t−0.275t^{-0.275}: it halves in the first 10 steps but then decays slowly, so a long run keeps a non-trivial amount of noise for thousands of steps. The noise is in absolute units, so whether σ=0.08\sigma=0.08 is large depends on the typical gradient norm of the model. That is one reason the useful η\eta varies across tasks.

Why temperature matters so much: the figure's barrier

  1. 1Barrier height from the shallow minimum: f(0.152)−f(1.332)=0.0226−(−0.5877)=0.610f(0.152)-f(1.332)=0.0226-(-0.5877)=0.610.
  2. 2Arrhenius factor eΔf/Te^{\Delta f/T}: e0.610/0.3=7.6e^{0.610/0.3}=7.6 at T=0.3T=0.3, but e0.610/0.1=447e^{0.610/0.1}=447 at T=0.1T=0.1. Cooling by a factor of 3 makes escape about 59 times slower.
  3. 3That matches the figure: at T=0.3T=0.3, 85% of the particles have crossed by step 1000; at T=0.1T=0.1, only 4.5%.
  4. 4Going back out of the deep well needs Δf=0.0226−(−1.435)=1.458\Delta f=0.0226-(-1.435)=1.458, so e1.458/0.3≈129e^{1.458/0.3}\approx129: 17 times harder than leaving the shallow well at the same temperature. This asymmetry is why most particles end in the deeper basin.
python
1import torch
2
3def add_gradient_noise(params, step, eta=0.3, gamma=0.55):
4 """g <- g + N(0, eta / (1 + step)^gamma), in place, before opt.step()."""
5 std = (eta / (1 + step) ** gamma) ** 0.5
6 for p in params:
7 if p.grad is not None:
8 p.grad.add_(torch.randn_like(p.grad), alpha=std)
9
10# for step, (x, y) in enumerate(loader):
11# opt.zero_grad(set_to_none=True)
12# loss_fn(model(x), y).backward()
13# add_gradient_noise(model.parameters(), step)
14# opt.step()
15
16@torch.no_grad()
17def sgld_step(params, lr, temperature=1.0):
18 """theta <- theta - lr * grad + sqrt(2 * lr * T) * xi.
19 For posterior sampling the loss must be the full negative log-posterior:
20 N * mean_batch_NLL + (-log prior), not the mean loss."""
21 for p in params:
22 if p.grad is None:
23 continue
24 p.add_(p.grad, alpha=-lr)
25 p.add_(torch.randn_like(p), alpha=(2 * lr * temperature) ** 0.5)
Annealed gradient noise (Neelakantan et al.) and an SGLD step, in plain PyTorch

SGD already has noise

A mini-batch gradient is the full gradient plus noise with covariance Σ(θ)/B\Sigma(\theta)/B. Plain SGD therefore already behaves like Langevin dynamics at an effective temperature of order α tr⁡Σ/B\alpha\,\operatorname{tr}\Sigma/B (Mandt, Hoffman and Blei, 2017; see Stochastic gradient descent (SGD) and Mini-batch gradient descent, where the noise scale α/B\alpha/B appears). Two differences matter. SGD's noise is anisotropic: near a minimum its covariance is aligned with the curvature, large along sharp directions and small along flat ones. And it vanishes on examples the model already fits. Isotropic Gaussian noise of the same size spends most of its variance in directions that do not matter.

Zhu, Wu, Yu, Wu and Ma (2019) showed that this structure is what lets SGD escape sharp minima: in their experiments, isotropic noise of comparable magnitude did not reproduce SGD's regularization effect, while noise with the gradient covariance's structure did. This also explains why adding Gaussian noise to large-batch training is not a reliable substitute for a small batch.

Escaping saddles

In high dimensions most stationary points of a non-convex loss are saddles, not minima. Exact gradient descent started exactly on a saddle's stable manifold never leaves (the figure's stationary-point start shows the one-dimensional version). From a random initialization GD almost surely avoids strict saddles (Lee et al., 2016), but it can take exponentially long to get past a sequence of them (Du et al., 2017). Noise fixes this: Ge, Huang, Jin and Yuan (2015) showed that noisy SGD escapes strict saddles in polynomial time, and Jin et al. (2017) showed that a single small isotropic perturbation, added whenever the gradient is small, escapes with only a polylogarithmic dependence on the dimension. This is the cleanest theoretical case for gradient noise.

Use when

  • +Two lines of code; no extra forward or backward passes
  • +Escapes saddles and flat regions that exact GD can stall on
  • +Annealing gives early exploration and late convergence
  • +With the Langevin scaling, the same update samples a posterior (SGLD)

Avoid when

  • −Isotropic noise is a poor match for the loss geometry; SGD's own anisotropic noise is usually more useful
  • −Little or no benefit on well-initialized, normalized, residual networks
  • −Absolute-scale hyperparameter η that must be tuned per model; interacts with Adam and clipping
  • −Polynomial cooling gives no global-optimum guarantee

Check yourself

You train with learning rate α = 0.01 and add gradient noise of standard deviation σ = 2. In the Langevin picture, what temperature does this correspond to?

Check yourself

In the figure, constant noise (γ = 0) puts about 60% of the particles in the deep basin, yet their mean loss is worse than with γ = 0.55. Why?

Remember

  • ✓Update θ←θ−α(gt+ϵt)\theta\leftarrow\theta-\alpha(g_t+\epsilon_t) with ϵt∼N(0,σt2I)\epsilon_t\sim\mathcal N(0,\sigma_t^2I) and σt2=η/(1+t)γ\sigma_t^2=\eta/(1+t)^\gamma (Neelakantan et al., 2015: γ=0.55\gamma=0.55).
  • ✓It is Langevin dynamics at temperature Tt=ασt2/2T_t=\alpha\sigma_t^2/2: a cooling schedule that explores early and settles late; escape time grows like eΔf/Te^{\Delta f/T}.
  • ✓With T = 1, a decreasing step and the log-posterior as loss, the same update is SGLD, a posterior sampler (Welling and Teh, 2011).
  • ✓SGD already injects anisotropic, curvature-aligned noise of scale ~α/B; isotropic Gaussian noise is usually a weaker regularizer than that.
  • ✓Clear theoretical benefit for escaping saddles; mixed empirical evidence on modern, well-tuned networks.
← back to Deep Learning roadmap