Law of Large Numbers & the CLT

Sample means converge to the true mean, and their distribution approaches a Gaussian with spread σ/√n, whatever the original distribution.

6 min read309 wordscore3 viewsupdated 2026-10-03

Definition

The law of large numbers (LLN) and the central limit theorem (CLT) describe the sample mean Xˉn=1n∑i=1nXi\bar X_n=\frac1n\sum_{i=1}^nX_i of nn independent draws. The LLN says Xˉn→μ\bar X_n\to\mu. The CLT says the error Xˉn−μ\bar X_n-\mu is approximately Gaussian, with standard deviation σ/n\sigma/\sqrt n.

At a glance

Setting
X1,…,XnX_1,\dots,X_n i.i.d. with mean μ\mu and finite variance σ2\sigma^2
LLN
Xˉn→μ\bar X_n\to\mu as n→∞n\to\infty
CLT
n (Xˉn−μ)/σ→N(0,1)\sqrt n\,(\bar X_n-\mu)/\sigma\to\mathcal N(0,1) in distribution
Standard error
SE⁡(Xˉn)=σ/n\operatorname{SE}(\bar X_n)=\sigma/\sqrt n — four times the data halves the error
Requires
independence (or weak dependence) and finite variance
Used in
confidence intervals, A/B tests, Monte Carlo estimates, mini-batch SGD

The intuition

Averaging cancels noise. Each draw misses μ\mu by a random amount; in a sum, positive and negative misses partly cancel, so the average wanders less as nn grows. And whatever shape single draws have — flat, skewed, U-shaped, discrete — the average of many takes the same bell shape. That universality is why Gaussian error bars work for data that is not itself Gaussian.

The math

Linearity of expectation and independence give the mean and variance of Xˉn\bar X_n directly:

E[Xˉn]=μ,Var⁡(Xˉn)=1n2∑i=1nVar⁡(Xi)=σ2n\mathbb E[\bar X_n]=\mu,\qquad \operatorname{Var}(\bar X_n)=\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(X_i)=\frac{\sigma^2}{n}

The spread shrinks like 1/n1/\sqrt n, and Chebyshev's inequality turns that into the weak law of large numbers:

P(∣Xˉn−μ∣≥ε)≤σ2n ε2 →n→∞ 0for every ε>0P\big(|\bar X_n-\mu|\ge\varepsilon\big)\le\frac{\sigma^2}{n\,\varepsilon^2}\ \xrightarrow[n\to\infty]{}\ 0\quad\text{for every }\varepsilon>0

The CLT describes the shape of that shrinking spread:

Xˉn−μσ/n →d N(0,1),i.e.Xˉn≈N(μ, σ2n)\frac{\bar X_n-\mu}{\sigma/\sqrt n}\ \xrightarrow{d}\ \mathcal N(0,1),\qquad\text{i.e.}\quad \bar X_n\approx\mathcal N\Big(\mu,\ \frac{\sigma^2}{n}\Big)
→d\xrightarrow{d}
convergence in distribution: the CDFs converge
σ/n\sigma/\sqrt n
standard error of the mean

How large must nn be? It depends on skew. Symmetric distributions look Gaussian by n≈10n\approx10; strongly skewed ones (exponential, incomes) can need hundreds. The Berry–Esseen theorem bounds the worst-case CDF error by Cρ/(σ3n)C\rho/(\sigma^3\sqrt n) with ρ=E∣X−μ∣3\rho=\mathbb E|X-\mu|^3 and C<0.48C<0.48.

See it

source:μ = 1 · σ = 1.000
0240 means of n = 5— N(μ, σ²/n)one draw
σ/√n = 0.4472SD of means = —mean of means = —

Law of large numbers: running mean of one sequence of 600 draws (log x-axis), with ±2σ/√k band

μ110100600after 600: 0.985
fig.Each run draws 2,000 sample means. Exponential at n = 5 is still right-skewed; at n = 30 the means' SD is 0.185 against σ/√n = 0.183. The U-shaped source already turns into a hump at n = 2. Below: one running mean settling onto μ inside its ±2σ/√k band.

Worked example

A fair die has μ=3.5\mu=3.5, σ2=3512\sigma^2=\frac{35}{12}, so σ≈1.708\sigma\approx1.708. Average n=30n=30 rolls: SE⁡=1.708/30≈0.312\operatorname{SE}=1.708/\sqrt{30}\approx0.312. The CLT predicts about 95% of such averages fall in 3.5±1.96⋅0.312=[2.89, 4.11]3.5\pm1.96\cdot0.312=[2.89,\ 4.11]. The exact probability, by convolving 30 dice, is 95.25%.

In code

python
1import numpy as np
2
3rng = np.random.default_rng(0)
4n, reps = 30, 100_000
5means = rng.integers(1, 7, size=(reps, n)).mean(axis=1)
6
7means.mean(), means.std() # ≈ 3.5, ≈ 0.312 = sqrt(35/12)/sqrt(30)
8np.mean(np.abs(means - 3.5) < 1.96 * 0.312) # ≈ 0.95
Simulate 100,000 averages of 30 die rolls.

Where it's used

Error bars

xˉ±1.96 s/n\bar x\pm1.96\,s/\sqrt n is a valid 95% interval for non-normal data, because of the CLT.

A/B testing

Differences in conversion rates are approximately Gaussian, so z-tests apply.

Monte Carlo

Estimate an expectation by averaging simulations; the error falls as 1/n1/\sqrt n — 100× the samples for one more correct digit.

Mini-batch SGD

A mini-batch gradient is a sample mean; its noise falls as 1/B1/\sqrt B in the batch size BB.

Check yourself

To halve the standard error of a sample mean you need…

Check yourself

X∼Exponential(1)X\sim\text{Exponential}(1) has μ=1\mu=1, σ=1\sigma=1. The mean of 100 draws is approximately (writing N(mean,variance)\mathcal N(\text{mean},\text{variance}))…

Remember

  • ✓E[Xˉn]=μ\mathbb E[\bar X_n]=\mu and SE⁡=σ/n\operatorname{SE}=\sigma/\sqrt n.
  • ✓LLN: the sample mean converges to the true mean.
  • ✓CLT: Xˉn≈N(μ,σ2/n)\bar X_n\approx\mathcal N(\mu,\sigma^2/n) for any finite-variance distribution.
  • ✓Needs independence and finite variance; describes means, not individual values.
← back to Mathematics roadmap