# Confidence Intervals

> A 95% confidence interval comes from a procedure that captures the true parameter in 95% of repeated samples — not a 95% chance for this one interval.

Source: https://www.mlfoundry.in/learn/confidence-intervals  
Section: Mathematics › Statistics  
Level: core · 6 min read · updated 2026-10-03

**A confidence interval (CI):** at level $1-\alpha$ is a range $[L(X),U(X)]$ computed from the data by a procedure that contains the true parameter $\theta$ with probability $1-\alpha$ **over repeated samples**: $P\big(L(X)\le\theta\le U(X)\big)=1-\alpha$.

- **Notation:** $\bar x\pm t^*_{n-1}\,s/\sqrt n$ for a mean
- **Input → output:** a sample and a confidence level → an interval
- **What is random:** the interval endpoints — not the parameter $\theta$
- **Key property:** width $\propto 1/\sqrt n$: four times the data halves the width
- **Prerequisites:** sampling distributions, the central limit theorem, the t-distribution
- **Used in:** error bars, A/B test reports, uncertainty of model metrics

## The intuition

A point estimate such as $\bar x=52$ hides how much it would change with a different sample. A CI reports a range of plausible values, and its width shows the precision. The 95% describes the *method*: repeat the study many times, and 95% of the intervals built this way contain the true value.

## The math

For a mean, the standardized estimate follows a Student t-distribution with $n-1$ degrees of freedom when the data are normal — and approximately otherwise, by the central limit theorem:

$$
\frac{\bar X-\mu}{S/\sqrt n}\sim t_{n-1}
$$

- $S$: sample standard deviation (divides by $n-1$)
- $S/\sqrt n$: standard error (SE) of the mean

Choose $t^*$ so the middle $1-\alpha$ of $t_{n-1}$ lies in $[-t^*,t^*]$, and solve the inequality for $\mu$:

$$
\bar x\ \pm\ t^*_{n-1,\,1-\alpha/2}\cdot\frac{s}{\sqrt n}
$$

As $n$ grows, $t^*\to z^*=1.96$ at 95%. For a proportion, the normal approximation is $\hat p\pm z^*\sqrt{\hat p(1-\hat p)/n}$; with small counts use the Wilson interval instead.

Three levers set the width: the confidence level (through $t^*$), the spread of the data ($s$), and the sample size ($\sqrt n$).

## See it

> Figure: Each bar is one sample's interval; red bars miss μ = 50. Coverage settles at the nominal level — 95.2% over 1,000 samples at n = 10, 95% level. Raise n to shrink the bars; lower the level to see more misses.

## Worked example

$n=9$, $\bar x=52$, $s=6$. The standard error is $6/\sqrt9=2$. With 8 degrees of freedom, $t^*=2.306$, so the 95% CI is $52\pm4.61=[47.39,\ 56.61]$. Using $z^*=1.96$ instead gives $[48.08,\ 55.92]$ — too narrow for $n=9$.

## In code

```python
import numpy as np
from scipy import stats

x = np.array([48.1, 55.3, 51.0, 47.6, 58.2, 50.4, 53.9, 46.8, 56.7])
m, se = x.mean(), stats.sem(x)                    # sem = s / sqrt(n)
lo, hi = stats.t.interval(0.95, df=len(x) - 1, loc=m, scale=se)
```

A t-interval for a mean. For proportions, statsmodels' proportion_confint(method='wilson').

- **A/B tests:** Report the lift with its interval — “+2.1% [0.4%, 3.8%]” shows both significance and size.
- **Model evaluation:** 90% accuracy on 500 test examples has a 95% CI of about $\pm2.6$ points; smaller differences between models are noise.
- **Forecasting:** Prediction intervals for a new observation are wider than confidence intervals for the mean — they add the noise variance.
- **Product analytics:** Margins of error on rates and averages decide whether a metric actually moved.

> **Not a probability for this interval:** Once computed, $[47.39, 56.61]$ either contains $\mu$ or it doesn't. The 95% belongs to the procedure. “$\mu$ lies in this interval with 95% probability” is a Bayesian credible interval, which needs a prior.

> **Overlap is not ‘no difference’:** Two 95% CIs can overlap while the difference between the groups is still significant. Compute an interval for the difference itself.

> **Why t and not z:** With small $n$, $s$ is itself noisy, so the standardized mean has heavier tails than a Gaussian. In the simulation, using $z^*=1.96$ at $n=3$ covers $\mu$ only **80%** of the time; the correct $t^*=4.30$ restores 95%.

**Question:** To halve the width of a confidence interval for a mean, you need about…

- A. 2× the data
- B. 4× the data
- C. 8× the data
- D. the same data at 90% confidence

**Answer:** B. 4× the data — Width $\propto s/\sqrt n$; halving it requires $\sqrt{n'}=2\sqrt n$, i.e. $n'=4n$.

**Question:** A 95% CI for a conversion rate is [2.1%, 2.9%]. Which statement is correct?

- A. There is a 95% probability the true rate is in [2.1%, 2.9%]
- B. 95% of users convert at a rate between 2.1% and 2.9%
- C. Intervals built this way contain the true rate in 95% of repeated samples
- D. The sample rate falls in this interval 95% of the time

**Answer:** C. Intervals built this way contain the true rate in 95% of repeated samples — The parameter is fixed; the interval is random. Coverage is a long-run property of the procedure.

**Key points**

- Mean: $\bar x\pm t^*_{n-1}\,s/\sqrt n$; $t^*\to1.96$ for large $n$ at 95%.
- The confidence level is a property of the procedure over repeated samples.
- Width $\propto1/\sqrt n$: four times the data halves it.
- Use t for small n; compare groups with an interval for the difference.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
