Confidence Intervals

A 95% confidence interval comes from a procedure that captures the true parameter in 95% of repeated samples — not a 95% chance for this one interval.

6 min read352 wordscore6 viewsupdated 2026-10-03

Definition

A confidence interval (CI) at level 1−α1-\alpha is a range [L(X),U(X)][L(X),U(X)] computed from the data by a procedure that contains the true parameter θ\theta with probability 1−α1-\alpha over repeated samples: P(L(X)≤θ≤U(X))=1−αP\big(L(X)\le\theta\le U(X)\big)=1-\alpha.

At a glance

Notation
xˉ±tn−1∗ s/n\bar x\pm t^*_{n-1}\,s/\sqrt n for a mean
Input → output
a sample and a confidence level → an interval
What is random
the interval endpoints — not the parameter θ\theta
Key property
width ∝1/n\propto 1/\sqrt n: four times the data halves the width
Prerequisites
sampling distributions, the central limit theorem, the t-distribution
Used in
error bars, A/B test reports, uncertainty of model metrics

The intuition

A point estimate such as xˉ=52\bar x=52 hides how much it would change with a different sample. A CI reports a range of plausible values, and its width shows the precision. The 95% describes the method: repeat the study many times, and 95% of the intervals built this way contain the true value.

The math

For a mean, the standardized estimate follows a Student t-distribution with n−1n-1 degrees of freedom when the data are normal — and approximately otherwise, by the central limit theorem:

Xˉ−μS/n∼tn−1\frac{\bar X-\mu}{S/\sqrt n}\sim t_{n-1}
SS
sample standard deviation (divides by n−1n-1)
S/nS/\sqrt n
standard error (SE) of the mean

Choose t∗t^* so the middle 1−α1-\alpha of tn−1t_{n-1} lies in [−t∗,t∗][-t^*,t^*], and solve the inequality for μ\mu:

xˉ ± tn−1, 1−α/2∗⋅sn\bar x\ \pm\ t^*_{n-1,\,1-\alpha/2}\cdot\frac{s}{\sqrt n}

As nn grows, t∗→z∗=1.96t^*\to z^*=1.96 at 95%. For a proportion, the normal approximation is p^±z∗p^(1−p^)/n\hat p\pm z^*\sqrt{\hat p(1-\hat p)/n}; with small counts use the Wilson interval instead.

Three levers set the width: the confidence level (through t∗t^*), the spread of the data (ss), and the sample size (n\sqrt n).

See it

newest sample on top · each bar = x̄ ± t·s/√ntrue μ = 5020304050607080
samples 0contain μ 0.0%nominal 95%t* = 2.262 · width 0.0
fig.Each bar is one sample's interval; red bars miss μ = 50. Coverage settles at the nominal level — 95.2% over 1,000 samples at n = 10, 95% level. Raise n to shrink the bars; lower the level to see more misses.

Worked example

n=9n=9, xˉ=52\bar x=52, s=6s=6. The standard error is 6/9=26/\sqrt9=2. With 8 degrees of freedom, t∗=2.306t^*=2.306, so the 95% CI is 52±4.61=[47.39, 56.61]52\pm4.61=[47.39,\ 56.61]. Using z∗=1.96z^*=1.96 instead gives [48.08, 55.92][48.08,\ 55.92] — too narrow for n=9n=9.

In code

python
1import numpy as np
2from scipy import stats
3
4x = np.array([48.1, 55.3, 51.0, 47.6, 58.2, 50.4, 53.9, 46.8, 56.7])
5m, se = x.mean(), stats.sem(x) # sem = s / sqrt(n)
6lo, hi = stats.t.interval(0.95, df=len(x) - 1, loc=m, scale=se)
A t-interval for a mean. For proportions, statsmodels' proportion_confint(method='wilson').

Where it's used

A/B tests

Report the lift with its interval — “+2.1% [0.4%, 3.8%]” shows both significance and size.

Model evaluation

90% accuracy on 500 test examples has a 95% CI of about ±2.6\pm2.6 points; smaller differences between models are noise.

Forecasting

Prediction intervals for a new observation are wider than confidence intervals for the mean — they add the noise variance.

Product analytics

Margins of error on rates and averages decide whether a metric actually moved.

Check yourself

To halve the width of a confidence interval for a mean, you need about…

Check yourself

A 95% CI for a conversion rate is [2.1%, 2.9%]. Which statement is correct?

Remember

  • ✓Mean: xˉ±tn−1∗ s/n\bar x\pm t^*_{n-1}\,s/\sqrt n; t∗→1.96t^*\to1.96 for large nn at 95%.
  • ✓The confidence level is a property of the procedure over repeated samples.
  • ✓Width ∝1/n\propto1/\sqrt n: four times the data halves it.
  • ✓Use t for small n; compare groups with an interval for the difference.
← back to Mathematics roadmap