Definition
A hypothesis test decides whether data are inconsistent with a null hypothesis (e.g. “no difference”). It computes a test statistic and the p-value ; if , it rejects .
At a glance
- Notation
- , , significance level , statistic , p-value
- Input → output
- data and a null model → a p-value and a reject / don't-reject decision
- Errors
- Type I (false positive) rate ; Type II (false negative) rate ; power
- Key property
- under , is uniform on — so
- Prerequisites
- sampling distributions, the Gaussian, the central limit theorem
- Used in
- A/B tests, model comparison, feature screening, drift detection
The intuition
Assume nothing interesting is happening — the null. Ask how surprising your data would be in that world. If data this extreme would almost never occur under the null, the null is a poor explanation. A small p-value measures surprise under ; it does not measure how likely is.
The math
The procedure
- 1State and , and fix (often 0.05) before looking at the data.
- 2Choose a statistic whose distribution under is known, e.g. .
- 3Compute : the probability under of a statistic at least as extreme as the one observed.
- 4Reject if ; otherwise fail to reject — which is not evidence that is true.
For a z-statistic:
- standard normal CDF
- observed statistic
Power. If the true effect is standard deviations, the statistic is centred at instead of 0, and the probability of rejecting (one-sided) is
The right-hand form gives the sample size to collect in advance. Common tests: one- and two-sample t-tests (means), the χ² test (counts in a contingency table), the two-proportion z-test (conversion rates), and permutation tests (any statistic, minimal assumptions).
See it
Worked example
with known , , . Then . Two-sided : reject at , not at . One-sided, .
Sample size. To detect with 80% power at one-sided : , so 39 observations (power at : 80.3%).
In code
1import numpy as np2from scipy import stats3from statsmodels.stats.proportion import proportions_ztest45rng = np.random.default_rng(0)6a = rng.normal(10.0, 2.0, 50) # control7b = rng.normal(10.8, 2.0, 50) # treatment8t, p = stats.ttest_ind(a, b, equal_var=False) # Welch's two-sample t-test910z, p_conv = proportions_ztest([120, 150], [2400, 2400]) # 5.0% vs 6.25%Where it's used
A/B testing
Did the new checkout raise conversion? A two-proportion test, with n fixed in advance by a power calculation.
Model comparison
McNemar's test, or a paired test across cross-validation folds, checks whether model B beats model A beyond noise.
Feature screening
χ² or ANOVA tests rank candidate features — with a multiple-comparison correction.
Drift monitoring
Kolmogorov–Smirnov or χ² tests flag when live inputs no longer match the training data.
Check yourself
A test returns p = 0.03. Which statement is correct?
Check yourself
With α held fixed, which change increases power?
Remember
- ✓ — surprise under the null, not .
- ✓Reject when ; is the false-positive rate you accept.
- ✓Power ; choose before collecting data.
- ✓Correct for multiple tests; always report effect sizes with intervals.