# The Bootstrap

> Resample your data with replacement many times and recompute the statistic; the spread of the results estimates its sampling uncertainty.

Source: https://www.mlfoundry.in/learn/the-bootstrap  
Section: Mathematics › Statistics  
Level: core · 5 min read · updated 2026-10-03

**The bootstrap:** estimates the sampling distribution of a statistic $\hat\theta$ by treating the sample as a stand-in for the population: draw $B$ resamples of size $n$ **with replacement**, compute $\hat\theta^*_b$ on each, and use the spread of $\hat\theta^*_1,\dots,\hat\theta^*_B$ as the uncertainty of $\hat\theta$.

- **Notation:** resample $x^*$, replicate $\hat\theta^*_b$, number of resamples $B$
- **Input → output:** one sample and any statistic → standard error and confidence interval
- **Cost:** $B$ recomputations of the statistic, typically 1,000–10,000
- **Key property:** needs no formula for the statistic's variance
- **Assumes:** independent, identically distributed units, and a sample that represents the population
- **Used in:** intervals for medians, AUC and other metrics; bagging and random forests

## The intuition

Uncertainty means: how much would $\hat\theta$ change if we collected a new sample? We can't collect one. The sample is our best picture of the population, so we sample *from the sample*. Drawing with replacement produces datasets that differ the way real new samples would — some points repeat, others are left out.

## The math

The empirical distribution $\hat F$ puts mass $1/n$ on each observation. The bootstrap replaces the unknown population $F$ with $\hat F$ and measures the spread of the statistic across resamples:

$$
\widehat{\operatorname{SE}}_{\text{boot}}=\sqrt{\frac{1}{B-1}\sum_{b=1}^{B}\big(\hat\theta^*_b-\bar\theta^*\big)^2},\qquad \bar\theta^*=\frac1B\sum_{b=1}^{B}\hat\theta^*_b
$$

- $\hat\theta^*_b$: statistic computed on resample $b$
- $B$: number of resamples

The **percentile interval** takes the 2.5th and 97.5th percentiles of the $\hat\theta^*_b$ as a 95% CI. Each resample leaves out a given point with probability $(1-\frac1n)^n\approx e^{-1}=36.8\%$ — the same out-of-bag fraction random forests use.

**Algorithm**

1. Compute $\hat\theta$ on the original sample of size $n$.
2. Draw $n$ indices uniformly **with replacement** to form a resample $x^*$.
3. Compute $\hat\theta^*_b$ on $x^*$; repeat for $b=1,\dots,B$.
4. SE = standard deviation of the $\hat\theta^*_b$; 95% CI = their 2.5th and 97.5th percentiles.

## See it

> Figure: Each resample draws 15 values with replacement (stacked dots = repeats). After 1,000 resamples the median is 22.1 with SE 4.63 and 95% CI [16.8, 34.0] — asymmetric, as skewed data should give. For the mean: bootstrap SE 3.00 vs the formula s/√n = 3.12.

## Worked example

Sample $\{3, 5, 10\}$, mean 6. There are $3^3=27$ equally likely ordered resamples, so the bootstrap SE of the mean can be computed exactly: it is $\hat\sigma/\sqrt n$ with the divide-by-$n$ standard deviation $\hat\sigma=\sqrt{26/3}=2.94$, giving $1.70$. The textbook $s/\sqrt n$ divides by $n-1$ and gives $2.08$ — at tiny $n$ the bootstrap is slightly optimistic.

## In code

```python
import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
x = np.array([10.8, 11.2, 11.5, 14.5, 16.8, 18.3, 19.2, 22.1,
              24.5, 25.7, 34.0, 36.5, 38.5, 43.7, 46.4])

B = 10_000
idx = rng.integers(0, len(x), size=(B, len(x)))   # B resamples at once
reps = np.median(x[idx], axis=1)
se = reps.std(ddof=1)
lo, hi = np.percentile(reps, [2.5, 97.5])

stats.bootstrap((x,), np.median, confidence_level=0.95)   # BCa by default
```

Vectorized percentile bootstrap of the median, and SciPy's BCa interval.

- **Metric uncertainty:** Intervals for AUC, F1 or accuracy on a fixed test set: resample test examples, recompute the metric.
- **Medians & quantiles:** p95 latency, median income — statistics with no simple standard-error formula.
- **Skewed A/B metrics:** Revenue per user is heavy-tailed: bootstrap the difference in means or medians between arms.
- **Bagging:** Random forests train each tree on a bootstrap sample; out-of-bag points estimate test error.

> **Resample the independent unit:** Rows must be independent. For time series, resample contiguous blocks (block bootstrap); for users with many events, resample users, not events — otherwise the interval is far too narrow.

> **It cannot create information:** With tiny samples or extreme statistics, the bootstrap is a poor stand-in: the maximum of a resample can never exceed the sample maximum. Percentile intervals also undercover at small n; BCa intervals correct for bias and skew.

**Question:** In one bootstrap resample of n = 1,000 rows, about how many distinct original rows appear?

- A. 1,000
- B. ≈ 632
- C. ≈ 500
- D. ≈ 368

**Answer:** B. ≈ 632 — Each row is missed with probability $(1-\frac1n)^n\approx e^{-1}\approx0.368$, so about $63.2\%$ appear at least once.

**Question:** You bootstrap a model's AUC on 10,000 test events from 500 users (20 events each), resampling events individually. The interval will be…

- A. about right
- B. too wide
- C. too narrow
- D. biased upward but the right width

**Answer:** C. too narrow — Events from the same user are correlated. Resampling them as if independent understates the variance; resample users instead.

**Key points**

- Resample n rows with replacement B times; recompute the statistic each time.
- SE = standard deviation of the replicates; 95% CI = 2.5th–97.5th percentiles (or BCa).
- Works for any statistic — medians, AUC, ratios — without a formula.
- Requires independent units: resample users or blocks, not correlated rows.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
