Hypothesis Testing & p-values

A p-value is the probability, if the null hypothesis were true, of a result at least as extreme as the one observed.

7 min read415 wordscore2 viewsupdated 2026-10-03

Definition

A hypothesis test decides whether data are inconsistent with a null hypothesis H0H_0 (e.g. “no difference”). It computes a test statistic TT and the p-value p=P(T at least as extreme as tobs∣H0)p=P(T\text{ at least as extreme as }t_{\text{obs}}\mid H_0); if p≤αp\le\alpha, it rejects H0H_0.

At a glance

Notation
H0H_0, H1H_1, significance level α\alpha, statistic TT, p-value pp
Input → output
data and a null model → a p-value and a reject / don't-reject decision
Errors
Type I (false positive) rate =α=\alpha; Type II (false negative) rate =β=\beta; power =1−β=1-\beta
Key property
under H0H_0, pp is uniform on [0,1][0,1] — so P(p≤α∣H0)=αP(p\le\alpha\mid H_0)=\alpha
Prerequisites
sampling distributions, the Gaussian, the central limit theorem
Used in
A/B tests, model comparison, feature screening, drift detection

The intuition

Assume nothing interesting is happening — the null. Ask how surprising your data would be in that world. If data this extreme would almost never occur under the null, the null is a poor explanation. A small p-value measures surprise under H0H_0; it does not measure how likely H0H_0 is.

The math

The procedure

  1. 1State H0H_0 and H1H_1, and fix α\alpha (often 0.05) before looking at the data.
  2. 2Choose a statistic whose distribution under H0H_0 is known, e.g. z=xˉ−μ0σ/n∼N(0,1)z=\frac{\bar x-\mu_0}{\sigma/\sqrt n}\sim\mathcal N(0,1).
  3. 3Compute pp: the probability under H0H_0 of a statistic at least as extreme as the one observed.
  4. 4Reject H0H_0 if p≤αp\le\alpha; otherwise fail to reject — which is not evidence that H0H_0 is true.

For a z-statistic:

pone-sided=1−Φ(zobs),ptwo-sided=2(1−Φ(∣zobs∣))p_{\text{one-sided}}=1-\Phi(z_{\text{obs}}),\qquad p_{\text{two-sided}}=2\big(1-\Phi(|z_{\text{obs}}|)\big)
Φ\Phi
standard normal CDF
zobsz_{\text{obs}}
observed statistic

Power. If the true effect is dd standard deviations, the statistic is centred at dnd\sqrt n instead of 0, and the probability of rejecting (one-sided) is

1−β=1−Φ ⁣(z1−α∗−dn)⟹n=(z1−α∗+z1−β∗d)21-\beta=1-\Phi\!\left(z^*_{1-\alpha}-d\sqrt n\right)\quad\Longrightarrow\quad n=\left(\frac{z^*_{1-\alpha}+z^*_{1-\beta}}{d}\right)^2

The right-hand form gives the sample size to collect in advance. Common tests: one- and two-sample t-tests (means), the χ² test (counts in a contingency table), the two-proportion z-test (conversion rates), and permutation tests (any statistic, minimal assumptions).

See it

z* = 1.64-4-3-2-1012345678H₀: Z ~ N(0, 1)H₁: N(1.79, 1)z = 2.10
p = 0.0179decision reject H₀power 1 − β = 55.7%β = 44.3%
αdrag the z marker
fig.Blue: the statistic's distribution if H₀ is true; orange: the p-value of the observed z (drag it). Dashed green: its distribution under the true effect — its mass beyond z* is the power. At d = 0.4, n = 20, one-sided α = 0.05, power is 55.7%.

Worked example

H0:μ=100H_0:\mu=100 with known σ=15\sigma=15, n=36n=36, xˉ=105\bar x=105. Then z=105−10015/36=2.0z=\frac{105-100}{15/\sqrt{36}}=2.0. Two-sided p=2(1−Φ(2))=0.046p=2(1-\Phi(2))=0.046: reject at α=0.05\alpha=0.05, not at α=0.01\alpha=0.01. One-sided, p=0.023p=0.023.

Sample size. To detect d=0.4d=0.4 with 80% power at one-sided α=0.05\alpha=0.05: n=(1.645+0.8420.4)2=38.6n=\left(\frac{1.645+0.842}{0.4}\right)^2=38.6, so 39 observations (power at n=39n=39: 80.3%).

In code

python
1import numpy as np
2from scipy import stats
3from statsmodels.stats.proportion import proportions_ztest
4
5rng = np.random.default_rng(0)
6a = rng.normal(10.0, 2.0, 50) # control
7b = rng.normal(10.8, 2.0, 50) # treatment
8t, p = stats.ttest_ind(a, b, equal_var=False) # Welch's two-sample t-test
9
10z, p_conv = proportions_ztest([120, 150], [2400, 2400]) # 5.0% vs 6.25%
Welch's t-test for means and a two-proportion z-test for conversion rates.

Where it's used

A/B testing

Did the new checkout raise conversion? A two-proportion test, with n fixed in advance by a power calculation.

Model comparison

McNemar's test, or a paired test across cross-validation folds, checks whether model B beats model A beyond noise.

Feature screening

χ² or ANOVA tests rank candidate features — with a multiple-comparison correction.

Drift monitoring

Kolmogorov–Smirnov or χ² tests flag when live inputs no longer match the training data.

Check yourself

A test returns p = 0.03. Which statement is correct?

Check yourself

With α held fixed, which change increases power?

Remember

  • ✓p=P(result at least this extreme∣H0)p=P(\text{result at least this extreme}\mid H_0) — surprise under the null, not P(H0)P(H_0).
  • ✓Reject when p≤αp\le\alpha; α\alpha is the false-positive rate you accept.
  • ✓Power =1−Φ(z∗−dn)=1-\Phi(z^*-d\sqrt n); choose nn before collecting data.
  • ✓Correct for multiple tests; always report effect sizes with intervals.
← back to Mathematics roadmap