# Law of Large Numbers & the CLT

> Sample means converge to the true mean, and their distribution approaches a Gaussian with spread σ/√n, whatever the original distribution.

Source: https://www.mlfoundry.in/learn/law-of-large-numbers-clt  
Section: Mathematics › Probability  
Level: core · 6 min read · updated 2026-10-03

**The law of large numbers (LLN) and the central limit theorem (CLT):** describe the sample mean $\bar X_n=\frac1n\sum_{i=1}^nX_i$ of $n$ independent draws. The **LLN** says $\bar X_n\to\mu$. The **CLT** says the error $\bar X_n-\mu$ is approximately Gaussian, with standard deviation $\sigma/\sqrt n$.

- **Setting:** $X_1,\dots,X_n$ i.i.d. with mean $\mu$ and finite variance $\sigma^2$
- **LLN:** $\bar X_n\to\mu$ as $n\to\infty$
- **CLT:** $\sqrt n\,(\bar X_n-\mu)/\sigma\to\mathcal N(0,1)$ in distribution
- **Standard error:** $\operatorname{SE}(\bar X_n)=\sigma/\sqrt n$ — four times the data halves the error
- **Requires:** independence (or weak dependence) and finite variance
- **Used in:** confidence intervals, A/B tests, Monte Carlo estimates, mini-batch SGD

## The intuition

Averaging cancels noise. Each draw misses $\mu$ by a random amount; in a sum, positive and negative misses partly cancel, so the average wanders less as $n$ grows. And whatever shape single draws have — flat, skewed, U-shaped, discrete — the average of many takes the same bell shape. That universality is why Gaussian error bars work for data that is not itself Gaussian.

## The math

Linearity of expectation and independence give the mean and variance of $\bar X_n$ directly:

$$
\mathbb E[\bar X_n]=\mu,\qquad \operatorname{Var}(\bar X_n)=\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(X_i)=\frac{\sigma^2}{n}
$$

The spread shrinks like $1/\sqrt n$, and Chebyshev's inequality turns that into the **weak law of large numbers**:

$$
P\big(|\bar X_n-\mu|\ge\varepsilon\big)\le\frac{\sigma^2}{n\,\varepsilon^2}\ \xrightarrow[n\to\infty]{}\ 0\quad\text{for every }\varepsilon>0
$$

The CLT describes the *shape* of that shrinking spread:

$$
\frac{\bar X_n-\mu}{\sigma/\sqrt n}\ \xrightarrow{d}\ \mathcal N(0,1),\qquad\text{i.e.}\quad \bar X_n\approx\mathcal N\Big(\mu,\ \frac{\sigma^2}{n}\Big)
$$

- $\xrightarrow{d}$: convergence in distribution: the CDFs converge
- $\sigma/\sqrt n$: standard error of the mean

How large must $n$ be? It depends on skew. Symmetric distributions look Gaussian by $n\approx10$; strongly skewed ones (exponential, incomes) can need hundreds. The Berry–Esseen theorem bounds the worst-case CDF error by $C\rho/(\sigma^3\sqrt n)$ with $\rho=\mathbb E|X-\mu|^3$ and $C<0.48$.

## See it

> Figure: Each run draws 2,000 sample means. Exponential at n = 5 is still right-skewed; at n = 30 the means' SD is 0.185 against σ/√n = 0.183. The U-shaped source already turns into a hump at n = 2. Below: one running mean settling onto μ inside its ±2σ/√k band.

## Worked example

A fair die has $\mu=3.5$, $\sigma^2=\frac{35}{12}$, so $\sigma\approx1.708$. Average $n=30$ rolls: $\operatorname{SE}=1.708/\sqrt{30}\approx0.312$. The CLT predicts about 95% of such averages fall in $3.5\pm1.96\cdot0.312=[2.89,\ 4.11]$. The exact probability, by convolving 30 dice, is 95.25%.

## In code

```python
import numpy as np

rng = np.random.default_rng(0)
n, reps = 30, 100_000
means = rng.integers(1, 7, size=(reps, n)).mean(axis=1)

means.mean(), means.std()                       # ≈ 3.5, ≈ 0.312 = sqrt(35/12)/sqrt(30)
np.mean(np.abs(means - 3.5) < 1.96 * 0.312)     # ≈ 0.95
```

Simulate 100,000 averages of 30 die rolls.

- **Error bars:** $\bar x\pm1.96\,s/\sqrt n$ is a valid 95% interval for non-normal data, because of the CLT.
- **A/B testing:** Differences in conversion rates are approximately Gaussian, so z-tests apply.
- **Monte Carlo:** Estimate an expectation by averaging simulations; the error falls as $1/\sqrt n$ — 100× the samples for one more correct digit.
- **Mini-batch SGD:** A mini-batch gradient is a sample mean; its noise falls as $1/\sqrt B$ in the batch size $B$.

> **The CLT is about means, not data:** Averages of 1,000 incomes are nearly Gaussian; the incomes themselves stay skewed. Don't use the CLT to justify $\pm2\sigma$ ranges for individual observations.

> **Independence and finite variance are required:** For the Cauchy distribution (no finite variance) the mean of $n$ draws has exactly the distribution of one draw — averaging never helps. Correlated samples (time series, many sessions per user) shrink much more slowly than $\sigma/\sqrt n$.

**Question:** To halve the standard error of a sample mean you need…

- A. 2× the data
- B. 4× the data
- C. √2× the data
- D. half the data

**Answer:** B. 4× the data — $\operatorname{SE}=\sigma/\sqrt n$; halving it needs $\sqrt{n'}=2\sqrt n$, i.e. $n'=4n$.

**Question:** $X\sim\text{Exponential}(1)$ has $\mu=1$, $\sigma=1$. The mean of 100 draws is approximately (writing $\mathcal N(\text{mean},\text{variance})$)…

- A. $\mathcal N(1,\ 1)$
- B. $\mathcal N(1,\ 0.01)$
- C. Exponential(1)
- D. $\mathcal N(1,\ 0.1)$

**Answer:** B. $\mathcal N(1,\ 0.01)$ — Variance $\sigma^2/n=1/100=0.01$, i.e. standard deviation 0.1.

**Key points**

- $\mathbb E[\bar X_n]=\mu$ and $\operatorname{SE}=\sigma/\sqrt n$.
- LLN: the sample mean converges to the true mean.
- CLT: $\bar X_n\approx\mathcal N(\mu,\sigma^2/n)$ for any finite-variance distribution.
- Needs independence and finite variance; describes means, not individual values.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
