# Hypothesis Testing & p-values

> A p-value is the probability, if the null hypothesis were true, of a result at least as extreme as the one observed.

Source: https://www.mlfoundry.in/learn/hypothesis-testing-p-values  
Section: Mathematics › Statistics  
Level: core · 7 min read · updated 2026-10-03

**A hypothesis test:** decides whether data are inconsistent with a **null hypothesis** $H_0$ (e.g. “no difference”). It computes a test statistic $T$ and the **p-value** $p=P(T\text{ at least as extreme as }t_{\text{obs}}\mid H_0)$; if $p\le\alpha$, it rejects $H_0$.

- **Notation:** $H_0$, $H_1$, significance level $\alpha$, statistic $T$, p-value $p$
- **Input → output:** data and a null model → a p-value and a reject / don't-reject decision
- **Errors:** Type I (false positive) rate $=\alpha$; Type II (false negative) rate $=\beta$; power $=1-\beta$
- **Key property:** under $H_0$, $p$ is uniform on $[0,1]$ — so $P(p\le\alpha\mid H_0)=\alpha$
- **Prerequisites:** sampling distributions, the Gaussian, the central limit theorem
- **Used in:** A/B tests, model comparison, feature screening, drift detection

## The intuition

Assume nothing interesting is happening — the null. Ask how surprising your data would be in that world. If data this extreme would almost never occur under the null, the null is a poor explanation. A small p-value measures **surprise under $H_0$**; it does not measure how likely $H_0$ is.

## The math

**The procedure**

1. State $H_0$ and $H_1$, and fix $\alpha$ (often 0.05) **before** looking at the data.
2. Choose a statistic whose distribution under $H_0$ is known, e.g. $z=\frac{\bar x-\mu_0}{\sigma/\sqrt n}\sim\mathcal N(0,1)$.
3. Compute $p$: the probability under $H_0$ of a statistic at least as extreme as the one observed.
4. Reject $H_0$ if $p\le\alpha$; otherwise *fail to reject* — which is not evidence that $H_0$ is true.

For a z-statistic:

$$
p_{\text{one-sided}}=1-\Phi(z_{\text{obs}}),\qquad p_{\text{two-sided}}=2\big(1-\Phi(|z_{\text{obs}}|)\big)
$$

- $\Phi$: standard normal CDF
- $z_{\text{obs}}$: observed statistic

**Power.** If the true effect is $d$ standard deviations, the statistic is centred at $d\sqrt n$ instead of 0, and the probability of rejecting (one-sided) is

$$
1-\beta=1-\Phi\!\left(z^*_{1-\alpha}-d\sqrt n\right)\quad\Longrightarrow\quad n=\left(\frac{z^*_{1-\alpha}+z^*_{1-\beta}}{d}\right)^2
$$

The right-hand form gives the sample size to collect *in advance*. Common tests: one- and two-sample **t-tests** (means), the **χ² test** (counts in a contingency table), the **two-proportion z-test** (conversion rates), and **permutation tests** (any statistic, minimal assumptions).

## See it

> Figure: Blue: the statistic's distribution if H₀ is true; orange: the p-value of the observed z (drag it). Dashed green: its distribution under the true effect — its mass beyond z* is the power. At d = 0.4, n = 20, one-sided α = 0.05, power is 55.7%.

## Worked example

$H_0:\mu=100$ with known $\sigma=15$, $n=36$, $\bar x=105$. Then $z=\frac{105-100}{15/\sqrt{36}}=2.0$. Two-sided $p=2(1-\Phi(2))=0.046$: reject at $\alpha=0.05$, not at $\alpha=0.01$. One-sided, $p=0.023$.

**Sample size.** To detect $d=0.4$ with 80% power at one-sided $\alpha=0.05$: $n=\left(\frac{1.645+0.842}{0.4}\right)^2=38.6$, so 39 observations (power at $n=39$: 80.3%).

## In code

```python
import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest

rng = np.random.default_rng(0)
a = rng.normal(10.0, 2.0, 50)                     # control
b = rng.normal(10.8, 2.0, 50)                     # treatment
t, p = stats.ttest_ind(a, b, equal_var=False)     # Welch's two-sample t-test

z, p_conv = proportions_ztest([120, 150], [2400, 2400])   # 5.0% vs 6.25%
```

Welch's t-test for means and a two-proportion z-test for conversion rates.

- **A/B testing:** Did the new checkout raise conversion? A two-proportion test, with n fixed in advance by a power calculation.
- **Model comparison:** McNemar's test, or a paired test across cross-validation folds, checks whether model B beats model A beyond noise.
- **Feature screening:** χ² or ANOVA tests rank candidate features — with a multiple-comparison correction.
- **Drift monitoring:** Kolmogorov–Smirnov or χ² tests flag when live inputs no longer match the training data.

> **p is not P(H₀ | data):** $p=0.03$ does not mean a 3% chance that the null is true, or a 97% chance that the effect is real. Those need a prior and Bayes' theorem — and when true effects are rare, most “significant” results can be false.

> **Multiple comparisons & p-hacking:** Run 20 independent tests on pure noise at $\alpha=0.05$ and $P(\text{at least one }p\le0.05)=1-0.95^{20}=64\%$. Testing many metrics, segments or stopping times inflates false positives. Fix the primary metric in advance, correct for multiplicity (Bonferroni $\alpha/m$, or Benjamini–Hochberg for the false discovery rate), and don't stop the moment $p$ dips below $\alpha$.

> **Significant ≠ important:** With $n=10^6$ users, a 0.01% lift can reach $p<10^{-6}$. Report the effect size with its confidence interval, and judge practical importance separately.

**Question:** A test returns p = 0.03. Which statement is correct?

- A. The null hypothesis has a 3% chance of being true
- B. If H₀ were true, a result at least this extreme would occur 3% of the time
- C. The effect is large
- D. The result will replicate with 97% probability

**Answer:** B. If H₀ were true, a result at least this extreme would occur 3% of the time — That is the definition: $p=P(T\ge t_{\text{obs}}\mid H_0)$. It says nothing directly about $P(H_0\mid\text{data})$ or effect size.

**Question:** With α held fixed, which change increases power?

- A. A smaller sample
- B. A larger true effect or a larger n
- C. Switching from one-sided to two-sided
- D. Lowering α from 0.05 to 0.01

**Answer:** B. A larger true effect or a larger n — Power $=1-\Phi(z^*-d\sqrt n)$ grows with $d\sqrt n$. Two-sided tests and smaller $\alpha$ raise $z^*$ and reduce power.

**Key points**

- $p=P(\text{result at least this extreme}\mid H_0)$ — surprise under the null, not $P(H_0)$.
- Reject when $p\le\alpha$; $\alpha$ is the false-positive rate you accept.
- Power $=1-\Phi(z^*-d\sqrt n)$; choose $n$ before collecting data.
- Correct for multiple tests; always report effect sizes with intervals.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
