Bayes' Theorem

P(H|E) = P(E|H)P(H) / P(E): posterior is proportional to likelihood times prior.

6 min read344 wordscore5 viewsupdated 2026-10-03

Definition

Bayes' theorem converts P(E∣H)P(E\mid H) — how likely the evidence is if a hypothesis holds — into P(H∣E)P(H\mid E) — how likely the hypothesis is given the evidence — by weighting with the prior P(H)P(H): posterior ∝ likelihood × prior.

At a glance

Statement
P(H∣E)=P(E∣H) P(H) / P(E)P(H\mid E)=P(E\mid H)\,P(H)\,/\,P(E)
Ingredients
prior P(H)P(H), likelihood P(E∣H)P(E\mid H), evidence P(E)=∑hP(E∣h)P(h)P(E)=\sum_h P(E\mid h)P(h)
Follows from
the definition of conditional probability, applied in both directions
Sequential use
today's posterior is tomorrow's prior
Prerequisites
conditional probability, the law of total probability
Used in
diagnostics, spam filters, Bayesian A/B tests, MAP estimation

The intuition

Evidence doesn't speak for itself; it multiplies belief by a factor. A positive test is strong evidence only if it is much more likely under sick than under healthy — and even strong evidence cannot overcome a tiny prior. Bayes makes this exact: multiply what you believed by how well each hypothesis predicted what you saw, then renormalize so the beliefs sum to 1.

The math

Write P(H∩E)P(H\cap E) both ways — P(H∣E)P(E)=P(E∣H)P(H)P(H\mid E)P(E)=P(E\mid H)P(H) — divide by P(E)P(E), and expand P(E)P(E) by the law of total probability:

P(H∣E)=P(E∣H) P(H)P(E∣H)P(H)+P(E∣¬H)P(¬H)P(H\mid E)=\frac{P(E\mid H)\,P(H)}{P(E\mid H)P(H)+P(E\mid\neg H)P(\neg H)}
P(H)P(H)
prior — belief before the evidence
P(E∣H)P(E\mid H)
likelihood — how well H predicts E
P(H∣E)P(H\mid E)
posterior — belief after the evidence

In odds form the normalizer cancels: posterior odds equal the likelihood ratio times the prior odds.

P(H∣E)P(¬H∣E)=P(E∣H)P(E∣¬H)⋅P(H)P(¬H)\frac{P(H\mid E)}{P(\neg H\mid E)}=\frac{P(E\mid H)}{P(E\mid\neg H)}\cdot\frac{P(H)}{P(\neg H)}

For parameters, replace HH by θ\theta and EE by data DD: p(θ∣D)∝p(D∣θ) p(θ)p(\theta\mid D)\propto p(D\mid\theta)\,p(\theta). For a coin with heads-probability θ\theta, a Beta(a,b)\mathrm{Beta}(a,b) prior and hh heads in nn flips give a Beta posterior — the prior acts like aa heads and bb tails already seen:

p(θ∣D) ∝ θh(1−θ)n−h⏟likelihood θa−1(1−θ)b−1⏟prior ⇒ θ∣D∼Beta(a+h, b+n−h)p(\theta\mid D)\ \propto\ \underbrace{\theta^{h}(1-\theta)^{n-h}}_{\text{likelihood}}\ \underbrace{\theta^{a-1}(1-\theta)^{b-1}}_{\text{prior}}\ \Rightarrow\ \theta\mid D\sim\mathrm{Beta}(a+h,\ b+n-h)

See it

■ sick, test + 4

■ healthy, test + 36

■ sick, test − 0

P(sick | +) =

9.2%

grid: 4 of 40 positives are sick

fig.Each square is one person out of 400. With 1% prevalence, most positive tests come from the large healthy group.
prior:
00.250.50.751true θ = 0.7- - prior · — posterior Beta(1, 1) · ▲ MLE
no flips yet — the posterior is the prior
flips 0 · heads 0 · tails 0MLE h/n = —posterior mean (a+h)/(a+b+n) = 0.50095% credible interval = [0.025, 0.975]
θ is hidden from the model; it only sees the flips.
fig.A coin with hidden bias 0.7. From a uniform prior, 20 flips give Beta(16, 6) with 95% interval [0.53, 0.89]. A wrong Beta(2, 12) prior needs more flips, but after 200 its interval [0.62, 0.74] contains the truth.

Worked example

A disease affects 1% of people. A test detects it 90% of the time and false-alarms 9% of the time. After a positive test: P(D∣+)=0.9⋅0.010.9⋅0.01+0.09⋅0.99=0.0090.0981≈9.2%P(D\mid+)=\frac{0.9\cdot0.01}{0.9\cdot0.01+0.09\cdot0.99}=\frac{0.009}{0.0981}\approx9.2\%. In odds form: prior odds 1:991:99, likelihood ratio 0.9/0.09=100.9/0.09=10, posterior odds 10:9910:99, so P=10/109≈9.2%P=10/109\approx9.2\%.

In code

python
1from scipy.stats import beta
2
3def posterior(prior, sens, fpr):
4 evidence = sens * prior + fpr * (1 - prior)
5 return sens * prior / evidence
6
7posterior(0.01, 0.90, 0.09) # 0.0917
8
9# uniform Beta(1, 1) prior, 7 heads in 10 flips → Beta(8, 4)
10post = beta(1 + 7, 1 + 3)
11post.mean(), post.interval(0.95) # 0.667, (0.390, 0.891)
A discrete posterior, and a Beta posterior for a coin.

Where it's used

Medical diagnostics

Post-test probability from prevalence, sensitivity and specificity.

Spam filtering

Naive Bayes multiplies per-word likelihood ratios into posterior odds.

Bayesian A/B tests

Report P(B better than A∣data)P(\text{B better than A}\mid\text{data}) directly from Beta posteriors.

Regularization

MAP estimation with a Gaussian prior on weights is L2 regularization; a Laplace prior gives L1.

Check yourself

Same test (90% sensitivity, 9% false-positive rate), but prevalence is 50%. What is P(D | +)?

Check yourself

A Beta(2,2)\mathrm{Beta}(2,2) prior on a coin's bias, then 3 heads and 1 tail. The posterior mean is…

Remember

  • ✓Posterior ∝ likelihood × prior; divide by the evidence to normalize.
  • ✓Odds form: posterior odds = likelihood ratio × prior odds.
  • ✓Rare hypotheses need very strong evidence; P(E|H) is not P(H|E).
  • ✓Beta prior + coin flips → Beta posterior; priors on weights ≡ regularization.
← back to Mathematics roadmap