Batch gradient descent

Gradient descent with the exact gradient, averaged over all N training examples: one step is one epoch. It is deterministic and converges at the clean convex rates, but each step costs N gradient evaluations, so it suits small datasets and full-batch methods like L-BFGS.

9 min read877 wordscore0 viewsupdated 2026-10-11

Definition

Batch gradient descent (full-batch GD) applies the gradient-descent update with the exact gradient of the training loss, the average of the per-example gradients over the entire dataset: θ←θ−η 1N∑i=1N∇ℓi(θ)\theta\leftarrow\theta-\eta\,\frac1N\sum_{i=1}^N\nabla\ell_i(\theta). Every update reads every training example once, so one step equals one epoch.

In machine learning the objective is almost always an average over data, the empirical risk f(θ)=1N∑iℓ(xi,yi;θ)f(\theta)=\frac1N\sum_i\ell(x_i,y_i;\theta). "Batch" refers to using the whole batch of data for each gradient, as opposed to one example (SGD) or a random subset (mini-batch). The confusing modern habit of calling a mini-batch "a batch" came later; in the optimization literature, batch methods are the full-gradient ones.

Batch gradient descent is what classical numerical optimization assumes: Cauchy's method, conjugate gradients, quasi-Newton methods such as L-BFGS and line searches all need a gradient that is exact and repeatable. It is the baseline against which the stochastic methods are measured, and for small problems it is still the right tool.

The full gradient

∇f(θt)=1N∑i=1N∇θ ℓ(xi,yi;θt),θt+1=θt−η ∇f(θt)\nabla f(\theta_t)=\frac1N\sum_{i=1}^{N}\nabla_\theta\,\ell(x_i,y_i;\theta_t),\qquad \theta_{t+1}=\theta_t-\eta\,\nabla f(\theta_t)
NN
number of training examples; every one contributes to every step
ℓ(xi,yi;θ)\ell(x_i,y_i;\theta)
per-example loss, e.g. 12(θ⊤xi−yi)2\tfrac12(\theta^\top x_i-y_i)^2
η\eta
learning rate; the same curvature limit η<2/L\eta<2/L applies as in plain gradient descent

For linear least squares, ℓi=12(wxi+b−yi)2\ell_i=\tfrac12(w x_i+b-y_i)^2, the gradient is the residual-weighted average ∇f=1N∑iri (xi, 1)\nabla f=\frac1N\sum_i r_i\,(x_i,\,1) with ri=wxi+b−yir_i=wx_i+b-y_i, and the Hessian is the constant matrix H=1N∑i(xi,1)(xi,1)⊤H=\frac1N\sum_i (x_i,1)(x_i,1)^\top. So the curvature constants L=λmax⁡(H)L=\lambda_{\max}(H) and μ=λmin⁡(H)\mu=\lambda_{\min}(H) are properties of the features: raw, uncentred inputs give a badly conditioned HH, standardized ones give H≈IH\approx I.

Properties

Gradient
exact: 1N∑i∇ℓi\frac1N\sum_i\nabla\ell_i
Cost per step
NN per-example gradients (forward + backward over the whole dataset), O(Nd)O(Nd) for dd parameters
Steps per epoch
exactly 1
Randomness
none: the same start gives the same path, bit for bit up to floating-point reduction order
Monotone
yes for η<2/L\eta<2/L; the training loss never goes up
Rate, convex
f(θt)−f∗≤L∥θ0−θ∗∥2/(2t)f(\theta_t)-f^*\le L\lVert\theta_0-\theta^*\rVert^2/(2t) with η=1/L\eta=1/L
Rate, strongly convex
(1−μ/L)t(1-\mu/L)^t; epochs to accuracy ε\varepsilon: O(κlog⁡1ε)O(\kappa\log\frac1\varepsilon), total work O(Nκlog⁡1ε)O(N\kappa\log\frac1\varepsilon)
Converges to
the exact minimizer (or stationary point): no noise floor
all 200 points, every stepx ∈ [0, 4], y ≈ 0.8x + 1 + noisepath in (w, b), κ = 29.3◆ least-squares optimum θ*f − f* (log) per epoch1e-81e-61e-41e-211e2epoch 0 … 60— batch GD - - SGD ··· ρ²ᵗ
|N =|
epoch 60 = 60 steps = 12,000 per-example gradients · f − f* = 0.005 · ‖∇f‖ = 0.045
L = 6.55, μ = 0.223, ρ = max|1 − ηλ| = 0.969 · f − f* ÷10⁶ after 149 epochs · SGD at η = 0.1/L: 0.002
fig.Full-batch gradient descent fitting ŷ = wx + b to N points with x ∈ [0, 4]. Left: every point feeds every step. Middle: the path in parameter space. Right: excess loss f − f* per epoch (log), the guaranteed rate ρ²ᵗ (dotted), and one-example SGD with η = 0.1/L run for the same number of epochs (dashed). With raw x and N = 200 the bowl has κ = 29.3: at ηL = 0.9 the first epoch removes the steep direction (f − f* from 18.7 to 0.37), then the path crawls along the valley and f − f* is still 4.6·10⁻³ after 60 epochs (10⁻⁶ of the start takes 149). Standardize x and κ = 1: the same ηL reaches 10⁻⁶ in 4 epochs. Switch to N = 2000: each step now costs 2000 gradients, and one epoch of SGD reaches 6.2·10⁻⁴ while batch GD is at 0.32. Set ηL above 2 and the loss grows every epoch.

Deterministic, monotone, and exact

Because the gradient is exact, batch gradient descent inherits everything proven for gradient descent: monotone decrease for η<2/L\eta<2/L, the O(1/t)O(1/t) rate on convex problems, the linear rate (1−μ/L)t(1-\mu/L)^t on strongly convex ones, and convergence to the exact minimizer. There is no noise floor, no need to decay the learning rate, and the loss curve is smooth, so a rising loss always means a real problem (usually η\eta too large).

Exactness also unlocks tools that stochastic methods cannot use directly. A line search picks η\eta at every step by checking the actual decrease (Armijo condition f(θ−ηg)≤f(θ)−c η∥g∥2f(\theta-\eta g)\le f(\theta)-c\,\eta\lVert g\rVert^2), so you do not need to know LL. Quasi-Newton methods such as L-BFGS build a curvature model from differences of successive gradients; with noisy gradients those differences are dominated by noise. Full-batch L-BFGS is common for small models, physics-informed networks and style-transfer-type optimizations of a single input.

Cost per step, and why it does not scale

Each step costs a full pass over the data, and the number of steps needed depends on κ\kappa and the accuracy, not on NN. Total work is therefore O(Nκlog⁡1ε)O(N\kappa\log\frac1\varepsilon): double the dataset and every step costs twice as much, even if the extra data are near-copies of what you already had. Stochastic methods instead pay a cost per step that does not depend on NN.

Bottou and Bousquet (2008) made the trade-off precise: when the goal is low test error rather than an exactly minimized training loss, optimizing beyond the statistical accuracy of the data is wasted work, and SGD reaches a given generalization error with far less computation than full-batch methods on large datasets. On ImageNet (1.28 million training images), one batch step would require a forward and backward pass over the whole training set, the same work with which mini-batch SGD at batch size 256 takes about 5,000 steps.

Worked example: four points

Fit y^=wx+b\hat y=wx+b to x=(0,1,2,3)x=(0,1,2,3), y=(1,3,2,5)y=(1,3,2,5) with f=12⋅14∑i(wxi+b−yi)2f=\frac12\cdot\frac14\sum_i(wx_i+b-y_i)^2, starting from w=b=0w=b=0 with η=0.25\eta=0.25. The Hessian is H=(3.51.51.51)H=\begin{pmatrix}3.5&1.5\\1.5&1\end{pmatrix}, with eigenvalues L=4.20L=4.20 and μ=0.297\mu=0.297 (κ=14.1\kappa=14.1), so the stability limit is 2/L=0.4762/L=0.476 and η=0.25\eta=0.25 is just above 1/L=0.2381/L=0.238. The least-squares solution is w∗=b∗=1.1w^*=b^*=1.1 with f∗=0.3375f^*=0.3375.

step(w, b)residuals r = ŷ − y∇f = (mean r·x, mean r)f
0(0, 0)(−1, −3, −2, −5)(−5.500, −2.750)4.875
1(1.375, 0.688)(−0.31, −0.94, 1.44, −0.19)(0.344, 0.000)0.385
2(1.289, 0.688)(−0.31, −1.02, 1.27, −0.45)(0.043, −0.129)0.368
3(1.278, 0.720)(−0.28, −1.00, 1.28, −0.45)(0.054, −0.113)0.364
Batch gradient descent on four points, η = 0.25 (computed in Node)
  1. 1Step 1 uses all four residuals: ∂f/∂w=14(0⋅(−1)+1⋅(−3)+2⋅(−2)+3⋅(−5))=−5.5\partial f/\partial w=\frac14(0\cdot(-1)+1\cdot(-3)+2\cdot(-2)+3\cdot(-5))=-5.5 and ∂f/∂b=14(−11)=−2.75\partial f/\partial b=\frac14(-11)=-2.75. The update is (0,0)−0.25⋅(−5.5,−2.75)=(1.375, 0.6875)(0,0)-0.25\cdot(-5.5,-2.75)=(1.375,\,0.6875).
  2. 2The loss falls from 4.875 to 0.385 in one step: the steep direction (mostly ww, eigenvalue 4.2) is almost solved immediately.
  3. 3What is left, 0.385−0.3375≈0.0470.385-0.3375\approx0.047, lies along the flat direction (eigenvalue 0.297), which shrinks asymptotically by only (1−0.25⋅0.297)2≈0.86(1-0.25\cdot0.297)^2\approx0.86 per step. Steps 2 and 3 barely move: this is κ=14\kappa=14 at work.
  4. 4Each step touched 4 examples. With 4 million examples drawn from the same distribution the path would be almost the same, but each step would cost a million times more.
python
1import torch, torch.nn as nn
2from torch.utils.data import DataLoader, TensorDataset
3
4X = torch.tensor([[0.], [1.], [2.], [3.]]); y = torch.tensor([[1.], [3.], [2.], [5.]])
5model = nn.Linear(1, 1)
6with torch.no_grad():
7 model.weight.zero_(); model.bias.zero_()
8mse = nn.MSELoss() # mean of r², so its gradient is 2× the ½-scaled f above
9
10# 1) The whole dataset as one batch: plain gradient descent on the empirical risk
11opt = torch.optim.SGD(model.parameters(), lr=0.125) # 0.125 on MSE = 0.25 on ½·MSE
12for epoch in range(3):
13 opt.zero_grad()
14 loss = mse(model(X), y)
15 loss.backward()
16 opt.step()
17 print(epoch + 1, model.weight.item(), model.bias.item()) # 1.375, 0.6875 after step 1
18
19# 2) Too big for memory: accumulate the exact full gradient over chunks, then step once
20loader = DataLoader(TensorDataset(X, y), batch_size=2, shuffle=False)
21N = len(X)
22opt.zero_grad()
23for xb, yb in loader:
24 loss = nn.functional.mse_loss(model(xb), yb, reduction="sum") / N # weight by share of N
25 loss.backward() # .grad accumulates
26opt.step() # one full-batch step
27
28# 3) Full-batch L-BFGS: needs exact, repeatable gradients, re-evaluated in a closure
29lbfgs = torch.optim.LBFGS(model.parameters(), lr=1, max_iter=50, line_search_fn="strong_wolfe")
30def closure():
31 lbfgs.zero_grad()
32 loss = mse(model(X), y)
33 loss.backward()
34 return loss
35lbfgs.step(closure)
36print(model.weight.item(), model.bias.item()) # ≈ 1.1, 1.1 (the least-squares solution)
Full-batch training in PyTorch: one big batch, chunked accumulation when it does not fit, or full-batch L-BFGS

Use when

  • +Exact gradient: no noise floor, converges to the exact minimizer with a constant step
  • +Monotone, reproducible loss curves; every convex convergence guarantee applies
  • +Enables line searches, conjugate gradients and L-BFGS
  • +Parallelizes perfectly: the sum over examples splits across devices

Avoid when

  • −One step costs NN gradient evaluations; total work grows linearly with dataset size
  • −Very few updates per unit of compute on large datasets, so early progress is slow
  • −The whole dataset must be processed (or accumulated) before any parameter moves
  • −No gradient noise, so none of SGD's tendency to escape sharp regions and saddle points

Check yourself

You train with batch gradient descent for 30 epochs on N = 50,000 examples. How many parameter updates and how many per-example gradient evaluations is that?

Check yourself

Your full-batch GD training loss goes up between two epochs. Which explanation is most likely?

Remember

  • ✓Batch GD uses the exact gradient 1N∑i∇ℓi\frac1N\sum_i\nabla\ell_i; one step = one epoch = NN gradient evaluations.
  • ✓Deterministic and monotone for η<2/L\eta<2/L, with rates O(1/t)O(1/t) (convex) and (1−μ/L)t(1-\mu/L)^t (strongly convex), and no noise floor.
  • ✓Epochs needed scale with κ\kappa; standardizing features can turn hundreds of epochs into a few.
  • ✓Total cost O(Nκlog⁡1ε)O(N\kappa\log\frac1\varepsilon) grows with NN, which is why large-scale training uses SGD or mini-batches instead.
  • ✓Use it for small data, convex problems, line searches and full-batch L-BFGS.
← back to Deep Learning roadmap