Definition
Batch gradient descent (full-batch GD) applies the gradient-descent update with the exact gradient of the training loss, the average of the per-example gradients over the entire dataset: . Every update reads every training example once, so one step equals one epoch.
In machine learning the objective is almost always an average over data, the empirical risk . "Batch" refers to using the whole batch of data for each gradient, as opposed to one example (SGD) or a random subset (mini-batch). The confusing modern habit of calling a mini-batch "a batch" came later; in the optimization literature, batch methods are the full-gradient ones.
Batch gradient descent is what classical numerical optimization assumes: Cauchy's method, conjugate gradients, quasi-Newton methods such as L-BFGS and line searches all need a gradient that is exact and repeatable. It is the baseline against which the stochastic methods are measured, and for small problems it is still the right tool.
The full gradient
- number of training examples; every one contributes to every step
- per-example loss, e.g.
- learning rate; the same curvature limit applies as in plain gradient descent
For linear least squares, , the gradient is the residual-weighted average with , and the Hessian is the constant matrix . So the curvature constants and are properties of the features: raw, uncentred inputs give a badly conditioned , standardized ones give .
Properties
- Gradient
- exact:
- Cost per step
- per-example gradients (forward + backward over the whole dataset), for parameters
- Steps per epoch
- exactly 1
- Randomness
- none: the same start gives the same path, bit for bit up to floating-point reduction order
- Monotone
- yes for ; the training loss never goes up
- Rate, convex
- with
- Rate, strongly convex
- ; epochs to accuracy : , total work
- Converges to
- the exact minimizer (or stationary point): no noise floor
Deterministic, monotone, and exact
Because the gradient is exact, batch gradient descent inherits everything proven for gradient descent: monotone decrease for , the rate on convex problems, the linear rate on strongly convex ones, and convergence to the exact minimizer. There is no noise floor, no need to decay the learning rate, and the loss curve is smooth, so a rising loss always means a real problem (usually too large).
Exactness also unlocks tools that stochastic methods cannot use directly. A line search picks at every step by checking the actual decrease (Armijo condition ), so you do not need to know . Quasi-Newton methods such as L-BFGS build a curvature model from differences of successive gradients; with noisy gradients those differences are dominated by noise. Full-batch L-BFGS is common for small models, physics-informed networks and style-transfer-type optimizations of a single input.
Cost per step, and why it does not scale
Each step costs a full pass over the data, and the number of steps needed depends on and the accuracy, not on . Total work is therefore : double the dataset and every step costs twice as much, even if the extra data are near-copies of what you already had. Stochastic methods instead pay a cost per step that does not depend on .
Bottou and Bousquet (2008) made the trade-off precise: when the goal is low test error rather than an exactly minimized training loss, optimizing beyond the statistical accuracy of the data is wasted work, and SGD reaches a given generalization error with far less computation than full-batch methods on large datasets. On ImageNet (1.28 million training images), one batch step would require a forward and backward pass over the whole training set, the same work with which mini-batch SGD at batch size 256 takes about 5,000 steps.
Worked example: four points
Fit to , with , starting from with . The Hessian is , with eigenvalues and (), so the stability limit is and is just above . The least-squares solution is with .
| step | (w, b) | residuals r = ŷ − y | ∇f = (mean r·x, mean r) | f |
|---|---|---|---|---|
| 0 | (0, 0) | (−1, −3, −2, −5) | (−5.500, −2.750) | 4.875 |
| 1 | (1.375, 0.688) | (−0.31, −0.94, 1.44, −0.19) | (0.344, 0.000) | 0.385 |
| 2 | (1.289, 0.688) | (−0.31, −1.02, 1.27, −0.45) | (0.043, −0.129) | 0.368 |
| 3 | (1.278, 0.720) | (−0.28, −1.00, 1.28, −0.45) | (0.054, −0.113) | 0.364 |
- 1Step 1 uses all four residuals: and . The update is .
- 2The loss falls from 4.875 to 0.385 in one step: the steep direction (mostly , eigenvalue 4.2) is almost solved immediately.
- 3What is left, , lies along the flat direction (eigenvalue 0.297), which shrinks asymptotically by only per step. Steps 2 and 3 barely move: this is at work.
- 4Each step touched 4 examples. With 4 million examples drawn from the same distribution the path would be almost the same, but each step would cost a million times more.
1import torch, torch.nn as nn2from torch.utils.data import DataLoader, TensorDataset34X = torch.tensor([[0.], [1.], [2.], [3.]]); y = torch.tensor([[1.], [3.], [2.], [5.]])5model = nn.Linear(1, 1)6with torch.no_grad():7 model.weight.zero_(); model.bias.zero_()8mse = nn.MSELoss() # mean of r², so its gradient is 2× the ½-scaled f above910# 1) The whole dataset as one batch: plain gradient descent on the empirical risk11opt = torch.optim.SGD(model.parameters(), lr=0.125) # 0.125 on MSE = 0.25 on ½·MSE12for epoch in range(3):13 opt.zero_grad()14 loss = mse(model(X), y)15 loss.backward()16 opt.step()17 print(epoch + 1, model.weight.item(), model.bias.item()) # 1.375, 0.6875 after step 11819# 2) Too big for memory: accumulate the exact full gradient over chunks, then step once20loader = DataLoader(TensorDataset(X, y), batch_size=2, shuffle=False)21N = len(X)22opt.zero_grad()23for xb, yb in loader:24 loss = nn.functional.mse_loss(model(xb), yb, reduction="sum") / N # weight by share of N25 loss.backward() # .grad accumulates26opt.step() # one full-batch step2728# 3) Full-batch L-BFGS: needs exact, repeatable gradients, re-evaluated in a closure29lbfgs = torch.optim.LBFGS(model.parameters(), lr=1, max_iter=50, line_search_fn="strong_wolfe")30def closure():31 lbfgs.zero_grad()32 loss = mse(model(X), y)33 loss.backward()34 return loss35lbfgs.step(closure)36print(model.weight.item(), model.bias.item()) # ≈ 1.1, 1.1 (the least-squares solution)Use when
- +Exact gradient: no noise floor, converges to the exact minimizer with a constant step
- +Monotone, reproducible loss curves; every convex convergence guarantee applies
- +Enables line searches, conjugate gradients and L-BFGS
- +Parallelizes perfectly: the sum over examples splits across devices
Avoid when
- −One step costs gradient evaluations; total work grows linearly with dataset size
- −Very few updates per unit of compute on large datasets, so early progress is slow
- −The whole dataset must be processed (or accumulated) before any parameter moves
- −No gradient noise, so none of SGD's tendency to escape sharp regions and saddle points
Check yourself
You train with batch gradient descent for 30 epochs on N = 50,000 examples. How many parameter updates and how many per-example gradient evaluations is that?
Check yourself
Your full-batch GD training loss goes up between two epochs. Which explanation is most likely?
Remember
- ✓Batch GD uses the exact gradient ; one step = one epoch = gradient evaluations.
- ✓Deterministic and monotone for , with rates (convex) and (strongly convex), and no noise floor.
- ✓Epochs needed scale with ; standardizing features can turn hundreds of epochs into a few.
- ✓Total cost grows with , which is why large-scale training uses SGD or mini-batches instead.
- ✓Use it for small data, convex problems, line searches and full-batch L-BFGS.