# Bayes' Theorem

> P(H|E) = P(E|H)P(H) / P(E): posterior is proportional to likelihood times prior.

Source: https://www.mlfoundry.in/learn/bayes-theorem  
Section: Mathematics › Probability  
Level: core · 6 min read · updated 2026-10-03

**Bayes' theorem:** converts $P(E\mid H)$ — how likely the evidence is if a hypothesis holds — into $P(H\mid E)$ — how likely the hypothesis is given the evidence — by weighting with the prior $P(H)$: posterior ∝ likelihood × prior.

- **Statement:** $P(H\mid E)=P(E\mid H)\,P(H)\,/\,P(E)$
- **Ingredients:** prior $P(H)$, likelihood $P(E\mid H)$, evidence $P(E)=\sum_h P(E\mid h)P(h)$
- **Follows from:** the definition of conditional probability, applied in both directions
- **Sequential use:** today's posterior is tomorrow's prior
- **Prerequisites:** conditional probability, the law of total probability
- **Used in:** diagnostics, spam filters, Bayesian A/B tests, MAP estimation

## The intuition

Evidence doesn't speak for itself; it multiplies belief by a factor. A positive test is strong evidence only if it is much more likely under *sick* than under *healthy* — and even strong evidence cannot overcome a tiny prior. Bayes makes this exact: multiply what you believed by how well each hypothesis predicted what you saw, then renormalize so the beliefs sum to 1.

## The math

Write $P(H\cap E)$ both ways — $P(H\mid E)P(E)=P(E\mid H)P(H)$ — divide by $P(E)$, and expand $P(E)$ by the law of total probability:

$$
P(H\mid E)=\frac{P(E\mid H)\,P(H)}{P(E\mid H)P(H)+P(E\mid\neg H)P(\neg H)}
$$

- $P(H)$: prior — belief before the evidence
- $P(E\mid H)$: likelihood — how well H predicts E
- $P(H\mid E)$: posterior — belief after the evidence

In **odds form** the normalizer cancels: posterior odds equal the likelihood ratio times the prior odds.

$$
\frac{P(H\mid E)}{P(\neg H\mid E)}=\frac{P(E\mid H)}{P(E\mid\neg H)}\cdot\frac{P(H)}{P(\neg H)}
$$

For parameters, replace $H$ by $\theta$ and $E$ by data $D$: $p(\theta\mid D)\propto p(D\mid\theta)\,p(\theta)$. For a coin with heads-probability $\theta$, a $\mathrm{Beta}(a,b)$ prior and $h$ heads in $n$ flips give a Beta posterior — the prior acts like $a$ heads and $b$ tails already seen:

$$
p(\theta\mid D)\ \propto\ \underbrace{\theta^{h}(1-\theta)^{n-h}}_{\text{likelihood}}\ \underbrace{\theta^{a-1}(1-\theta)^{b-1}}_{\text{prior}}\ \Rightarrow\ \theta\mid D\sim\mathrm{Beta}(a+h,\ b+n-h)
$$

## See it

> Figure: Each square is one person out of 400. With 1% prevalence, most positive tests come from the large healthy group.

> Figure: A coin with hidden bias 0.7. From a uniform prior, 20 flips give Beta(16, 6) with 95% interval [0.53, 0.89]. A wrong Beta(2, 12) prior needs more flips, but after 200 its interval [0.62, 0.74] contains the truth.

## Worked example

A disease affects 1% of people. A test detects it 90% of the time and false-alarms 9% of the time. After a positive test: $P(D\mid+)=\frac{0.9\cdot0.01}{0.9\cdot0.01+0.09\cdot0.99}=\frac{0.009}{0.0981}\approx9.2\%$. In odds form: prior odds $1:99$, likelihood ratio $0.9/0.09=10$, posterior odds $10:99$, so $P=10/109\approx9.2\%$.

## In code

```python
from scipy.stats import beta

def posterior(prior, sens, fpr):
    evidence = sens * prior + fpr * (1 - prior)
    return sens * prior / evidence

posterior(0.01, 0.90, 0.09)          # 0.0917

# uniform Beta(1, 1) prior, 7 heads in 10 flips → Beta(8, 4)
post = beta(1 + 7, 1 + 3)
post.mean(), post.interval(0.95)     # 0.667, (0.390, 0.891)
```

A discrete posterior, and a Beta posterior for a coin.

- **Medical diagnostics:** Post-test probability from prevalence, sensitivity and specificity.
- **Spam filtering:** Naive Bayes multiplies per-word likelihood ratios into posterior odds.
- **Bayesian A/B tests:** Report $P(\text{B better than A}\mid\text{data})$ directly from Beta posteriors.
- **Regularization:** MAP estimation with a Gaussian prior on weights is L2 regularization; a Laplace prior gives L1.

> **Base-rate neglect:** Intuition anchors on the 90% and ignores the 1% prior. When a condition is rare, false positives from the large healthy group outnumber the true positives.

> **Prosecutor's fallacy:** $P(E\mid H)\ne P(H\mid E)$. "A random innocent person matches this DNA with probability 1 in a million" does not mean "the suspect is innocent with probability 1 in a million" — in a city of 10 million, about 10 innocent people match.

**Question:** Same test (90% sensitivity, 9% false-positive rate), but prevalence is 50%. What is P(D | +)?

- A. ≈ 9%
- B. ≈ 50%
- C. ≈ 91%
- D. ≈ 99%

**Answer:** C. ≈ 91% — $\frac{0.9\cdot0.5}{0.9\cdot0.5+0.09\cdot0.5}=\frac{0.45}{0.495}\approx0.91$. The same test means very different things at different base rates.

**Question:** A $\mathrm{Beta}(2,2)$ prior on a coin's bias, then 3 heads and 1 tail. The posterior mean is…

- A. 0.5
- B. 0.625
- C. 0.75
- D. 0.6

**Answer:** B. 0.625 — The posterior is $\mathrm{Beta}(2+3,\ 2+1)=\mathrm{Beta}(5,3)$, with mean $5/(5+3)=0.625$ — between the prior mean 0.5 and the MLE 0.75.

**Key points**

- Posterior ∝ likelihood × prior; divide by the evidence to normalize.
- Odds form: posterior odds = likelihood ratio × prior odds.
- Rare hypotheses need very strong evidence; P(E|H) is not P(H|E).
- Beta prior + coin flips → Beta posterior; priors on weights ≡ regularization.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
