Definition
The bootstrap estimates the sampling distribution of a statistic by treating the sample as a stand-in for the population: draw resamples of size with replacement, compute on each, and use the spread of as the uncertainty of .
At a glance
- Notation
- resample , replicate , number of resamples
- Input → output
- one sample and any statistic → standard error and confidence interval
- Cost
- recomputations of the statistic, typically 1,000–10,000
- Key property
- needs no formula for the statistic's variance
- Assumes
- independent, identically distributed units, and a sample that represents the population
- Used in
- intervals for medians, AUC and other metrics; bagging and random forests
The intuition
Uncertainty means: how much would change if we collected a new sample? We can't collect one. The sample is our best picture of the population, so we sample from the sample. Drawing with replacement produces datasets that differ the way real new samples would — some points repeat, others are left out.
The math
The empirical distribution puts mass on each observation. The bootstrap replaces the unknown population with and measures the spread of the statistic across resamples:
- statistic computed on resample
- number of resamples
The percentile interval takes the 2.5th and 97.5th percentiles of the as a 95% CI. Each resample leaves out a given point with probability — the same out-of-bag fraction random forests use.
Algorithm
- 1Compute on the original sample of size .
- 2Draw indices uniformly with replacement to form a resample .
- 3Compute on ; repeat for .
- 4SE = standard deviation of the ; 95% CI = their 2.5th and 97.5th percentiles.
See it
- 1x ← original sample, size n
- 2for b = 1 … B:
- 3 x* ← n draws from x, w/ repl.
- 4 θ*_b ← stat(x*)
- 5 add θ*_b to histogram
- 6CI ← 2.5% and 97.5% of θ*
- 7SE ← sd(θ*₁ … θ*_B)
Original sample: n = 15 skewed values. Its median is θ̂ = 22.1 — but how uncertain is that number?
Worked example
Sample , mean 6. There are equally likely ordered resamples, so the bootstrap SE of the mean can be computed exactly: it is with the divide-by- standard deviation , giving . The textbook divides by and gives — at tiny the bootstrap is slightly optimistic.
In code
1import numpy as np2from scipy import stats34rng = np.random.default_rng(0)5x = np.array([10.8, 11.2, 11.5, 14.5, 16.8, 18.3, 19.2, 22.1,6 24.5, 25.7, 34.0, 36.5, 38.5, 43.7, 46.4])78B = 10_0009idx = rng.integers(0, len(x), size=(B, len(x))) # B resamples at once10reps = np.median(x[idx], axis=1)11se = reps.std(ddof=1)12lo, hi = np.percentile(reps, [2.5, 97.5])1314stats.bootstrap((x,), np.median, confidence_level=0.95) # BCa by defaultWhere it's used
Metric uncertainty
Intervals for AUC, F1 or accuracy on a fixed test set: resample test examples, recompute the metric.
Medians & quantiles
p95 latency, median income — statistics with no simple standard-error formula.
Skewed A/B metrics
Revenue per user is heavy-tailed: bootstrap the difference in means or medians between arms.
Bagging
Random forests train each tree on a bootstrap sample; out-of-bag points estimate test error.
Check yourself
In one bootstrap resample of n = 1,000 rows, about how many distinct original rows appear?
Check yourself
You bootstrap a model's AUC on 10,000 test events from 500 users (20 events each), resampling events individually. The interval will be…
Remember
- ✓Resample n rows with replacement B times; recompute the statistic each time.
- ✓SE = standard deviation of the replicates; 95% CI = 2.5th–97.5th percentiles (or BCa).
- ✓Works for any statistic — medians, AUC, ratios — without a formula.
- ✓Requires independent units: resample users or blocks, not correlated rows.