Definition
A confidence interval (CI) at level is a range computed from the data by a procedure that contains the true parameter with probability over repeated samples: .
At a glance
- Notation
- for a mean
- Input → output
- a sample and a confidence level → an interval
- What is random
- the interval endpoints — not the parameter
- Key property
- width : four times the data halves the width
- Prerequisites
- sampling distributions, the central limit theorem, the t-distribution
- Used in
- error bars, A/B test reports, uncertainty of model metrics
The intuition
A point estimate such as hides how much it would change with a different sample. A CI reports a range of plausible values, and its width shows the precision. The 95% describes the method: repeat the study many times, and 95% of the intervals built this way contain the true value.
The math
For a mean, the standardized estimate follows a Student t-distribution with degrees of freedom when the data are normal — and approximately otherwise, by the central limit theorem:
- sample standard deviation (divides by )
- standard error (SE) of the mean
Choose so the middle of lies in , and solve the inequality for :
As grows, at 95%. For a proportion, the normal approximation is ; with small counts use the Wilson interval instead.
Three levers set the width: the confidence level (through ), the spread of the data (), and the sample size ().
See it
Worked example
, , . The standard error is . With 8 degrees of freedom, , so the 95% CI is . Using instead gives — too narrow for .
In code
1import numpy as np2from scipy import stats34x = np.array([48.1, 55.3, 51.0, 47.6, 58.2, 50.4, 53.9, 46.8, 56.7])5m, se = x.mean(), stats.sem(x) # sem = s / sqrt(n)6lo, hi = stats.t.interval(0.95, df=len(x) - 1, loc=m, scale=se)Where it's used
A/B tests
Report the lift with its interval — “+2.1% [0.4%, 3.8%]” shows both significance and size.
Model evaluation
90% accuracy on 500 test examples has a 95% CI of about points; smaller differences between models are noise.
Forecasting
Prediction intervals for a new observation are wider than confidence intervals for the mean — they add the noise variance.
Product analytics
Margins of error on rates and averages decide whether a metric actually moved.
Check yourself
To halve the width of a confidence interval for a mean, you need about…
Check yourself
A 95% CI for a conversion rate is [2.1%, 2.9%]. Which statement is correct?
Remember
- ✓Mean: ; for large at 95%.
- ✓The confidence level is a property of the procedure over repeated samples.
- ✓Width : four times the data halves it.
- ✓Use t for small n; compare groups with an interval for the difference.