Definition
The law of large numbers (LLN) and the central limit theorem (CLT) describe the sample mean of independent draws. The LLN says . The CLT says the error is approximately Gaussian, with standard deviation .
At a glance
- Setting
- i.i.d. with mean and finite variance
- LLN
- as
- CLT
- in distribution
- Standard error
- — four times the data halves the error
- Requires
- independence (or weak dependence) and finite variance
- Used in
- confidence intervals, A/B tests, Monte Carlo estimates, mini-batch SGD
The intuition
Averaging cancels noise. Each draw misses by a random amount; in a sum, positive and negative misses partly cancel, so the average wanders less as grows. And whatever shape single draws have — flat, skewed, U-shaped, discrete — the average of many takes the same bell shape. That universality is why Gaussian error bars work for data that is not itself Gaussian.
The math
Linearity of expectation and independence give the mean and variance of directly:
The spread shrinks like , and Chebyshev's inequality turns that into the weak law of large numbers:
The CLT describes the shape of that shrinking spread:
- convergence in distribution: the CDFs converge
- standard error of the mean
How large must be? It depends on skew. Symmetric distributions look Gaussian by ; strongly skewed ones (exponential, incomes) can need hundreds. The Berry–Esseen theorem bounds the worst-case CDF error by with and .
See it
Law of large numbers: running mean of one sequence of 600 draws (log x-axis), with ±2σ/√k band
Worked example
A fair die has , , so . Average rolls: . The CLT predicts about 95% of such averages fall in . The exact probability, by convolving 30 dice, is 95.25%.
In code
1import numpy as np23rng = np.random.default_rng(0)4n, reps = 30, 100_0005means = rng.integers(1, 7, size=(reps, n)).mean(axis=1)67means.mean(), means.std() # ≈ 3.5, ≈ 0.312 = sqrt(35/12)/sqrt(30)8np.mean(np.abs(means - 3.5) < 1.96 * 0.312) # ≈ 0.95Where it's used
Error bars
is a valid 95% interval for non-normal data, because of the CLT.
A/B testing
Differences in conversion rates are approximately Gaussian, so z-tests apply.
Monte Carlo
Estimate an expectation by averaging simulations; the error falls as — 100× the samples for one more correct digit.
Mini-batch SGD
A mini-batch gradient is a sample mean; its noise falls as in the batch size .
Check yourself
To halve the standard error of a sample mean you need…
Check yourself
has , . The mean of 100 draws is approximately (writing )…
Remember
- ✓ and .
- ✓LLN: the sample mean converges to the true mean.
- ✓CLT: for any finite-variance distribution.
- ✓Needs independence and finite variance; describes means, not individual values.