# Batch gradient descent

> Gradient descent with the exact gradient, averaged over all N training examples: one step is one epoch. It is deterministic and converges at the clean convex rates, but each step costs N gradient evaluations, so it suits small datasets and full-batch methods like L-BFGS.

Source: https://www.mlfoundry.in/learn/batch-gradient-descent  
Section: Deep Learning › Optimization Algorithms  
Level: core · 9 min read · updated 2026-10-11

**Batch gradient descent:** (full-batch GD) applies the gradient-descent update with the **exact** gradient of the training loss, the average of the per-example gradients over the entire dataset: $\theta\leftarrow\theta-\eta\,\frac1N\sum_{i=1}^N\nabla\ell_i(\theta)$. Every update reads every training example once, so one step equals one epoch.

In machine learning the objective is almost always an average over data, the **empirical risk** $f(\theta)=\frac1N\sum_i\ell(x_i,y_i;\theta)$. "Batch" refers to using the whole batch of data for each gradient, as opposed to one example (**SGD**) or a random subset (**mini-batch**). The confusing modern habit of calling a mini-batch "a batch" came later; in the optimization literature, batch methods are the full-gradient ones.

Batch gradient descent is what classical numerical optimization assumes: Cauchy's method, conjugate gradients, quasi-Newton methods such as L-BFGS and line searches all need a gradient that is exact and repeatable. It is the baseline against which the stochastic methods are measured, and for small problems it is still the right tool.

## The full gradient

$$
\nabla f(\theta_t)=\frac1N\sum_{i=1}^{N}\nabla_\theta\,\ell(x_i,y_i;\theta_t),\qquad \theta_{t+1}=\theta_t-\eta\,\nabla f(\theta_t)
$$

- $N$: number of training examples; every one contributes to every step
- $\ell(x_i,y_i;\theta)$: per-example loss, e.g. $\tfrac12(\theta^\top x_i-y_i)^2$
- $\eta$: learning rate; the same curvature limit $\eta<2/L$ applies as in plain gradient descent

For linear least squares, $\ell_i=\tfrac12(w x_i+b-y_i)^2$, the gradient is the residual-weighted average $\nabla f=\frac1N\sum_i r_i\,(x_i,\,1)$ with $r_i=wx_i+b-y_i$, and the Hessian is the constant matrix $H=\frac1N\sum_i (x_i,1)(x_i,1)^\top$. So the curvature constants $L=\lambda_{\max}(H)$ and $\mu=\lambda_{\min}(H)$ are properties of the **features**: raw, uncentred inputs give a badly conditioned $H$, standardized ones give $H\approx I$.

**Properties**

- **Gradient:** exact: $\frac1N\sum_i\nabla\ell_i$
- **Cost per step:** $N$ per-example gradients (forward + backward over the whole dataset), $O(Nd)$ for $d$ parameters
- **Steps per epoch:** exactly 1
- **Randomness:** none: the same start gives the same path, bit for bit up to floating-point reduction order
- **Monotone:** yes for $\eta<2/L$; the training loss never goes up
- **Rate, convex:** $f(\theta_t)-f^*\le L\lVert\theta_0-\theta^*\rVert^2/(2t)$ with $\eta=1/L$
- **Rate, strongly convex:** $(1-\mu/L)^t$; epochs to accuracy $\varepsilon$: $O(\kappa\log\frac1\varepsilon)$, total work $O(N\kappa\log\frac1\varepsilon)$
- **Converges to:** the exact minimizer (or stationary point): no noise floor

> Figure: Full-batch gradient descent fitting ŷ = wx + b to N points with x ∈ [0, 4]. Left: every point feeds every step. Middle: the path in parameter space. Right: excess loss f − f* per epoch (log), the guaranteed rate ρ²ᵗ (dotted), and one-example SGD with η = 0.1/L run for the same number of epochs (dashed). With raw x and N = 200 the bowl has κ = 29.3: at ηL = 0.9 the first epoch removes the steep direction (f − f* from 18.7 to 0.37), then the path crawls along the valley and f − f* is still 4.6·10⁻³ after 60 epochs (10⁻⁶ of the start takes 149). Standardize x and κ = 1: the same ηL reaches 10⁻⁶ in 4 epochs. Switch to N = 2000: each step now costs 2000 gradients, and one epoch of SGD reaches 6.2·10⁻⁴ while batch GD is at 0.32. Set ηL above 2 and the loss grows every epoch.

## Deterministic, monotone, and exact

Because the gradient is exact, batch gradient descent inherits everything proven for gradient descent: monotone decrease for $\eta<2/L$, the $O(1/t)$ rate on convex problems, the linear rate $(1-\mu/L)^t$ on strongly convex ones, and convergence to the exact minimizer. There is no noise floor, no need to decay the learning rate, and the loss curve is smooth, so a rising loss always means a real problem (usually $\eta$ too large).

Exactness also unlocks tools that stochastic methods cannot use directly. A **line search** picks $\eta$ at every step by checking the actual decrease (Armijo condition $f(\theta-\eta g)\le f(\theta)-c\,\eta\lVert g\rVert^2$), so you do not need to know $L$. **Quasi-Newton** methods such as L-BFGS build a curvature model from differences of successive gradients; with noisy gradients those differences are dominated by noise. Full-batch L-BFGS is common for small models, physics-informed networks and style-transfer-type optimizations of a single input.

## Cost per step, and why it does not scale

Each step costs a full pass over the data, and the number of steps needed depends on $\kappa$ and the accuracy, not on $N$. Total work is therefore $O(N\kappa\log\frac1\varepsilon)$: double the dataset and every step costs twice as much, even if the extra data are near-copies of what you already had. Stochastic methods instead pay a cost per step that does not depend on $N$.

Bottou and Bousquet (2008) made the trade-off precise: when the goal is low **test** error rather than an exactly minimized training loss, optimizing beyond the statistical accuracy of the data is wasted work, and SGD reaches a given generalization error with far less computation than full-batch methods on large datasets. On ImageNet (1.28 million training images), one batch step would require a forward and backward pass over the whole training set, the same work with which mini-batch SGD at batch size 256 takes about 5,000 steps.

> **When batch gradient descent is the right choice:** Small datasets that fit in memory (thousands of examples, not millions); convex or near-convex problems where you want the exact optimum (logistic regression, GLMs, classical ML solvers); full-batch L-BFGS for small networks and scientific ML; debugging, where a deterministic loss curve removes one source of confusion; and theory, where it is the noise-free reference.

## Worked example: four points

Fit $\hat y=wx+b$ to $x=(0,1,2,3)$, $y=(1,3,2,5)$ with $f=\frac12\cdot\frac14\sum_i(wx_i+b-y_i)^2$, starting from $w=b=0$ with $\eta=0.25$. The Hessian is $H=\begin{pmatrix}3.5&1.5\\1.5&1\end{pmatrix}$, with eigenvalues $L=4.20$ and $\mu=0.297$ ($\kappa=14.1$), so the stability limit is $2/L=0.476$ and $\eta=0.25$ is just above $1/L=0.238$. The least-squares solution is $w^*=b^*=1.1$ with $f^*=0.3375$.

Batch gradient descent on four points, η = 0.25 (computed in Node)

| step | (w, b) | residuals r = ŷ − y | ∇f = (mean r·x, mean r) | f |
| --- | --- | --- | --- | --- |
| 0 | (0, 0) | (−1, −3, −2, −5) | (−5.500, −2.750) | 4.875 |
| 1 | (1.375, 0.688) | (−0.31, −0.94, 1.44, −0.19) | (0.344, 0.000) | 0.385 |
| 2 | (1.289, 0.688) | (−0.31, −1.02, 1.27, −0.45) | (0.043, −0.129) | 0.368 |
| 3 | (1.278, 0.720) | (−0.28, −1.00, 1.28, −0.45) | (0.054, −0.113) | 0.364 |

1. Step 1 uses all four residuals: $\partial f/\partial w=\frac14(0\cdot(-1)+1\cdot(-3)+2\cdot(-2)+3\cdot(-5))=-5.5$ and $\partial f/\partial b=\frac14(-11)=-2.75$. The update is $(0,0)-0.25\cdot(-5.5,-2.75)=(1.375,\,0.6875)$.
2. The loss falls from 4.875 to 0.385 in one step: the steep direction (mostly $w$, eigenvalue 4.2) is almost solved immediately.
3. What is left, $0.385-0.3375\approx0.047$, lies along the flat direction (eigenvalue 0.297), which shrinks asymptotically by only $(1-0.25\cdot0.297)^2\approx0.86$ per step. Steps 2 and 3 barely move: this is $\kappa=14$ at work.
4. Each step touched 4 examples. With 4 million examples drawn from the same distribution the path would be almost the same, but each step would cost a million times more.

```python
import torch, torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

X = torch.tensor([[0.], [1.], [2.], [3.]]); y = torch.tensor([[1.], [3.], [2.], [5.]])
model = nn.Linear(1, 1)
with torch.no_grad():
    model.weight.zero_(); model.bias.zero_()
mse = nn.MSELoss()                     # mean of r², so its gradient is 2× the ½-scaled f above

# 1) The whole dataset as one batch: plain gradient descent on the empirical risk
opt = torch.optim.SGD(model.parameters(), lr=0.125)   # 0.125 on MSE = 0.25 on ½·MSE
for epoch in range(3):
    opt.zero_grad()
    loss = mse(model(X), y)
    loss.backward()
    opt.step()
    print(epoch + 1, model.weight.item(), model.bias.item())   # 1.375, 0.6875 after step 1

# 2) Too big for memory: accumulate the exact full gradient over chunks, then step once
loader = DataLoader(TensorDataset(X, y), batch_size=2, shuffle=False)
N = len(X)
opt.zero_grad()
for xb, yb in loader:
    loss = nn.functional.mse_loss(model(xb), yb, reduction="sum") / N   # weight by share of N
    loss.backward()                                                     # .grad accumulates
opt.step()                                                              # one full-batch step

# 3) Full-batch L-BFGS: needs exact, repeatable gradients, re-evaluated in a closure
lbfgs = torch.optim.LBFGS(model.parameters(), lr=1, max_iter=50, line_search_fn="strong_wolfe")
def closure():
    lbfgs.zero_grad()
    loss = mse(model(X), y)
    loss.backward()
    return loss
lbfgs.step(closure)
print(model.weight.item(), model.bias.item())   # ≈ 1.1, 1.1 (the least-squares solution)
```

Full-batch training in PyTorch: one big batch, chunked accumulation when it does not fit, or full-batch L-BFGS

> **Mean, not sum:** If you sum the per-example losses instead of averaging them, the gradient (and $L$) grows with $N$, and a learning rate that worked on a subset diverges on the full set. Use the mean, or divide by $N$ when you accumulate chunks with `reduction="sum"`. Averaging in chunks of unequal size without weighting by chunk size gives a biased gradient.

> **Exact gradient, same curvature problem:** Removing gradient noise does nothing about conditioning. In the figure, batch GD with raw features still needs 149 epochs for a $10^{-6}$ reduction because $\kappa=29$, and the same data standardized need 4. On a well-conditioned problem batch GD is excellent; on an ill-conditioned one it is slow however exact its gradient.

**Strengths**

- Exact gradient: no noise floor, converges to the exact minimizer with a constant step
- Monotone, reproducible loss curves; every convex convergence guarantee applies
- Enables line searches, conjugate gradients and L-BFGS
- Parallelizes perfectly: the sum over examples splits across devices

**Limitations**

- One step costs $N$ gradient evaluations; total work grows linearly with dataset size
- Very few updates per unit of compute on large datasets, so early progress is slow
- The whole dataset must be processed (or accumulated) before any parameter moves
- No gradient noise, so none of SGD's tendency to escape sharp regions and saddle points

**Question:** You train with batch gradient descent for 30 epochs on N = 50,000 examples. How many parameter updates and how many per-example gradient evaluations is that?

- A. 30 updates, 1.5 million gradients
- B. 1.5 million updates, 1.5 million gradients
- C. 30 updates, 30 gradients
- D. 50,000 updates, 30 gradients

**Answer:** A. 30 updates, 1.5 million gradients — Each batch step uses all $N$ examples, so one step is one epoch: 30 updates, each costing 50,000 per-example gradients. One-example SGD would make 1.5 million updates for the same compute.

**Question:** Your full-batch GD training loss goes up between two epochs. Which explanation is most likely?

- A. Normal gradient noise
- B. The learning rate exceeds 2/L for the current curvature
- C. The dataset was shuffled
- D. The model has reached the noise floor

**Answer:** B. The learning rate exceeds 2/L for the current curvature — Batch GD has no noise and no shuffling: for $\eta<2/L$ the descent lemma guarantees the loss decreases. A loss that rises means the step overshoots in the steepest direction (or a bug), so reduce $\eta$ or use a line search.

**Key points**

- Batch GD uses the exact gradient $\frac1N\sum_i\nabla\ell_i$; one step = one epoch = $N$ gradient evaluations.
- Deterministic and monotone for $\eta<2/L$, with rates $O(1/t)$ (convex) and $(1-\mu/L)^t$ (strongly convex), and no noise floor.
- Epochs needed scale with $\kappa$; standardizing features can turn hundreds of epochs into a few.
- Total cost $O(N\kappa\log\frac1\varepsilon)$ grows with $N$, which is why large-scale training uses SGD or mini-batches instead.
- Use it for small data, convex problems, line searches and full-batch L-BFGS.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
