The Bootstrap

Resample your data with replacement many times and recompute the statistic; the spread of the results estimates its sampling uncertainty.

5 min read378 wordscore2 viewsupdated 2026-10-03

Definition

The bootstrap estimates the sampling distribution of a statistic θ^\hat\theta by treating the sample as a stand-in for the population: draw BB resamples of size nn with replacement, compute θ^b∗\hat\theta^*_b on each, and use the spread of θ^1∗,…,θ^B∗\hat\theta^*_1,\dots,\hat\theta^*_B as the uncertainty of θ^\hat\theta.

At a glance

Notation
resample x∗x^*, replicate θ^b∗\hat\theta^*_b, number of resamples BB
Input → output
one sample and any statistic → standard error and confidence interval
Cost
BB recomputations of the statistic, typically 1,000–10,000
Key property
needs no formula for the statistic's variance
Assumes
independent, identically distributed units, and a sample that represents the population
Used in
intervals for medians, AUC and other metrics; bagging and random forests

The intuition

Uncertainty means: how much would θ^\hat\theta change if we collected a new sample? We can't collect one. The sample is our best picture of the population, so we sample from the sample. Drawing with replacement produces datasets that differ the way real new samples would — some points repeat, others are left out.

The math

The empirical distribution F^\hat F puts mass 1/n1/n on each observation. The bootstrap replaces the unknown population FF with F^\hat F and measures the spread of the statistic across resamples:

SE⁡^boot=1B−1∑b=1B(θ^b∗−θˉ∗)2,θˉ∗=1B∑b=1Bθ^b∗\widehat{\operatorname{SE}}_{\text{boot}}=\sqrt{\frac{1}{B-1}\sum_{b=1}^{B}\big(\hat\theta^*_b-\bar\theta^*\big)^2},\qquad \bar\theta^*=\frac1B\sum_{b=1}^{B}\hat\theta^*_b
θ^b∗\hat\theta^*_b
statistic computed on resample bb
BB
number of resamples

The percentile interval takes the 2.5th and 97.5th percentiles of the θ^b∗\hat\theta^*_b as a 95% CI. Each resample leaves out a given point with probability (1−1n)n≈e−1=36.8%(1-\frac1n)^n\approx e^{-1}=36.8\% — the same out-of-bag fraction random forests use.

Algorithm

  1. 1Compute θ^\hat\theta on the original sample of size nn.
  2. 2Draw nn indices uniformly with replacement to form a resample x∗x^*.
  3. 3Compute θ^b∗\hat\theta^*_b on x∗x^*; repeat for b=1,…,Bb=1,\dots,B.
  4. 4SE = standard deviation of the θ^b∗\hat\theta^*_b; 95% CI = their 2.5th and 97.5th percentiles.

See it

statistic
▸ bootstrap · medianstep 1/17
original sample0102030405060θ̂ = 22.1bootstrap medians θ* · B = 0112539
B = 0θ̂ = 22.1SE* = —
  1. 1x ← original sample, size n
  2. 2for b = 1 … B:
  3. 3 x* ← n draws from x, w/ repl.
  4. 4 θ*_b ← stat(x*)
  5. 5 add θ*_b to histogram
  6. 6CI ← 2.5% and 97.5% of θ*
  7. 7SE ← sd(θ*₁ … θ*_B)

Original sample: n = 15 skewed values. Its median is θ̂ = 22.1 — but how uncertain is that number?

fig.Each resample draws 15 values with replacement (stacked dots = repeats). After 1,000 resamples the median is 22.1 with SE 4.63 and 95% CI [16.8, 34.0] — asymmetric, as skewed data should give. For the mean: bootstrap SE 3.00 vs the formula s/√n = 3.12.

Worked example

Sample {3,5,10}\{3, 5, 10\}, mean 6. There are 33=273^3=27 equally likely ordered resamples, so the bootstrap SE of the mean can be computed exactly: it is σ^/n\hat\sigma/\sqrt n with the divide-by-nn standard deviation σ^=26/3=2.94\hat\sigma=\sqrt{26/3}=2.94, giving 1.701.70. The textbook s/ns/\sqrt n divides by n−1n-1 and gives 2.082.08 — at tiny nn the bootstrap is slightly optimistic.

In code

python
1import numpy as np
2from scipy import stats
3
4rng = np.random.default_rng(0)
5x = np.array([10.8, 11.2, 11.5, 14.5, 16.8, 18.3, 19.2, 22.1,
6 24.5, 25.7, 34.0, 36.5, 38.5, 43.7, 46.4])
7
8B = 10_000
9idx = rng.integers(0, len(x), size=(B, len(x))) # B resamples at once
10reps = np.median(x[idx], axis=1)
11se = reps.std(ddof=1)
12lo, hi = np.percentile(reps, [2.5, 97.5])
13
14stats.bootstrap((x,), np.median, confidence_level=0.95) # BCa by default
Vectorized percentile bootstrap of the median, and SciPy's BCa interval.

Where it's used

Metric uncertainty

Intervals for AUC, F1 or accuracy on a fixed test set: resample test examples, recompute the metric.

Medians & quantiles

p95 latency, median income — statistics with no simple standard-error formula.

Skewed A/B metrics

Revenue per user is heavy-tailed: bootstrap the difference in means or medians between arms.

Bagging

Random forests train each tree on a bootstrap sample; out-of-bag points estimate test error.

Check yourself

In one bootstrap resample of n = 1,000 rows, about how many distinct original rows appear?

Check yourself

You bootstrap a model's AUC on 10,000 test events from 500 users (20 events each), resampling events individually. The interval will be…

Remember

  • ✓Resample n rows with replacement B times; recompute the statistic each time.
  • ✓SE = standard deviation of the replicates; 95% CI = 2.5th–97.5th percentiles (or BCa).
  • ✓Works for any statistic — medians, AUC, ratios — without a formula.
  • ✓Requires independent units: resample users or blocks, not correlated rows.
← back to Mathematics roadmap