Naive Bayes

Assume features are independent given the class, so P(y|x) ∝ P(y) Πᵢ P(xᵢ|y) — training is just counting.

17 min read1640 wordscore8 viewsupdated 2026-10-07

Definition

Naive Bayes is a family of probabilistic classifiers that model how each class generates data — a prior P(y)P(y) and, for every feature separately, a distribution P(xj∣y)P(x_j\mid y) — and then use Bayes' rule to turn that model around into P(y∣x)P(y\mid x). The "naive" part is the assumption that the features are conditionally independent given the class, which lets the joint likelihood factor into a product of one-feature pieces that can each be estimated by counting.

At a glance

Task
Classification (binary or multi-class), with class probabilities
Input → output
feature vector xx → posterior P(y=k∣x)P(y=k\mid x) for every class; predict the largest
Learns
class priors P(y=k)P(y=k) and one small distribution per (feature, class): a count table, a Gaussian (μ,σ2)(\mu,\sigma^2), or word frequencies
Training
one pass over the data: O(n d)O(n\,d) — no optimisation, no iterations
Prediction
O(d K)O(d\,K): add dd log-likelihoods for each of KK classes
Hyperparameters
smoothing α\alpha (counts), "var_smoothing" (Gaussian), class priors, and the choice of per-feature distribution
Assumes
features independent given the class; each feature follows the chosen family (categorical, Gaussian, multinomial, Bernoulli)

The problem

Bayes' rule tells us exactly how to classify: pick the class with the largest P(y∣x)∝P(y) P(x∣y)P(y\mid x)\propto P(y)\,P(x\mid y). The trouble is P(x∣y)P(x\mid y) — the probability of an entire feature vector. With just 30 yes/no features there are 230≈1092^{30}\approx 10^9 possible vectors per class; we would need billions of examples to estimate their probabilities, and almost every new input would be a vector we have never seen, with estimated probability 0.

Discriminative models such as logistic regression dodge the problem by modelling P(y∣x)P(y\mid x) directly, but they need iterative optimisation and enough data to pin down a weight per feature. Naive Bayes takes the opposite route: keep the generative story, but make one bold simplification so that P(x∣y)P(x\mid y) needs only dd one-dimensional estimates instead of one dd-dimensional one.

The idea

Pretend that, once you know the class, the features tell you nothing about each other. Knowing an email is spam, whether it contains "free" says nothing extra about whether it contains "money". Then the probability of the whole vector is just the product of the individual feature probabilities, and each of those is estimated from the training rows of that class — usually by counting.

To classify, every class starts with its prior and then each feature votes: its likelihood under that class multiplies in. In log space the votes add up, so naive Bayes is an evidence ledger — each feature contributes a fixed amount of log-odds for or against each class, and the class with the largest total wins.

Vocabulary

Prior
P(y=k)P(y=k): how common class kk is before looking at the features — the fraction of training rows with label kk
Likelihood
P(xj∣y=k)P(x_j\mid y=k): how probable feature value xjx_j is among examples of class kk
Posterior
P(y=k∣x)P(y=k\mid x): the class probability after seeing the features; what we predict from
Evidence
P(x)=∑kP(y=k)P(x∣y=k)P(x)=\sum_k P(y=k)P(x\mid y=k): the normaliser that makes posteriors sum to 1; it is the same for every class, so the arg-max ignores it
Conditional independence
P(xi,xj∣y)=P(xi∣y)P(xj∣y)P(x_i,x_j\mid y)=P(x_i\mid y)P(x_j\mid y): independence within each class. Features can still be strongly correlated overall (through the class)
Generative model
one that models P(x,y)=P(y)P(x∣y)P(x,y)=P(y)P(x\mid y) — how data is produced — rather than only the boundary P(y∣x)P(y\mid x)
Smoothing (α)
pseudo-counts added to every count so that an unseen value gets a small probability instead of zero
Event model
the per-feature distribution family: categorical, Gaussian, multinomial (word counts) or Bernoulli (word present/absent)

The math

Start from Bayes' rule for a class kk and a feature vector x=(x1,…,xd)x=(x_1,\dots,x_d):

P(y=k∣x)=P(y=k) P(x∣y=k)P(x)P(y=k\mid x)=\frac{P(y=k)\,P(x\mid y=k)}{P(x)}
P(y=k)P(y=k)
prior of class kk
P(x∣y=k)P(x\mid y=k)
class-conditional likelihood of the whole vector xx
P(x)P(x)
evidence ∑k′P(y=k′)P(x∣y=k′)\sum_{k'}P(y=k')P(x\mid y=k'), identical for all classes

The chain rule gives P(x∣y)=P(x1∣y) P(x2∣y,x1)⋯P(xd∣y,x1,…,xd−1)P(x\mid y)=P(x_1\mid y)\,P(x_2\mid y,x_1)\cdots P(x_d\mid y,x_1,\dots,x_{d-1}) exactly. The naive assumption drops every conditioning variable except the class, P(xj∣y,x1,…,xj−1)≈P(xj∣y)P(x_j\mid y,x_1,\dots,x_{j-1})\approx P(x_j\mid y), so

P(y=k∣x)  ∝  πk∏j=1dP(xj∣y=k),y^=arg⁡max⁡k[log⁡πk+∑j=1dlog⁡P(xj∣y=k)]P(y=k\mid x)\;\propto\;\pi_k\prod_{j=1}^{d}P(x_j\mid y=k),\qquad \hat y=\arg\max_k\Big[\log\pi_k+\sum_{j=1}^{d}\log P(x_j\mid y=k)\Big]
πk\pi_k
prior P(y=k)P(y=k), estimated as nk/nn_k/n
dd
number of features
y^\hat y
predicted class (the maximum a posteriori class)

We take logs for two reasons: the product of hundreds of small probabilities underflows to 0 in floating point, and sums are cheaper and easier to read than products. Posteriors are recovered with the log-sum-exp trick: subtract the largest class score, exponentiate, normalise.

Training is counting: the maximum-likelihood estimate

For a categorical feature jj with values v=1,…,Vjv=1,\dots,V_j, the model for class kk is a probability vector θkjv=P(xj=v∣y=k)\theta_{kjv}=P(x_j=v\mid y=k). Let ckjvc_{kjv} be the number of class-kk rows whose feature jj equals vv. The log-likelihood of the training data that involves this vector is ∑vckjvlog⁡θkjv\sum_v c_{kjv}\log\theta_{kjv}, to be maximised subject to ∑vθkjv=1\sum_v\theta_{kjv}=1. With a Lagrange multiplier λ\lambda:

∂∂θkjv[∑vckjvlog⁡θkjv−λ(∑vθkjv−1)]=ckjvθkjv−λ=0  ⇒  θ^kjv=ckjvnk\frac{\partial}{\partial\theta_{kjv}}\Big[\sum_{v}c_{kjv}\log\theta_{kjv}-\lambda\Big(\sum_v\theta_{kjv}-1\Big)\Big]=\frac{c_{kjv}}{\theta_{kjv}}-\lambda=0\;\Rightarrow\;\hat\theta_{kjv}=\frac{c_{kjv}}{n_k}
ckjvc_{kjv}
count of class-kk rows with xj=vx_j=v
nkn_k
number of class-kk rows; the constraint forces λ=∑vckjv=nk\lambda=\sum_v c_{kjv}=n_k
θ^kjv\hat\theta_{kjv}
estimated P(xj=v∣y=k)P(x_j=v\mid y=k)

So the maximum-likelihood fit is literally a frequency table, and because the log-likelihood splits into one independent term per (class, feature), every table is fitted on its own in a single pass. The same argument gives π^k=nk/n\hat\pi_k=n_k/n.

Smoothing. If a value never occurs with class kk, θ^kjv=0\hat\theta_{kjv}=0 and any input with that value gets posterior 0 for class kk, no matter how strongly every other feature points to it. The fix adds α\alpha pseudo-counts to every cell:

θ^kjv=ckjv+αnk+αVj\hat\theta_{kjv}=\frac{c_{kjv}+\alpha}{n_k+\alpha V_j}
α\alpha
smoothing strength: α=1\alpha=1 is Laplace smoothing, 0<α<10<\alpha<1 Lidstone; α=0\alpha=0 is the raw MLE
VjV_j
number of distinct values of feature jj (so the probabilities still sum to 1)

This is the posterior mean of θ\theta under a symmetric Dirichlet(α\alpha) prior: α\alpha imaginary observations of each value. Mathematically α\alpha shrinks every estimate toward the uniform 1/Vj1/V_j, and the shrinkage weight is αVj/(nk+αVj)\alpha V_j/(n_k+\alpha V_j) — large when the class is rare, negligible when nk≫αVjn_k\gg\alpha V_j. Larger α\alpha therefore means flatter likelihood ratios, less extreme posteriors, and more bias; smaller α\alpha means more variance.

Continuous features: Gaussian naive Bayes

For a real-valued feature, the usual choice is one Gaussian per (class, feature). The maximum-likelihood estimates are again closed-form — the class's sample mean and (biased, divide-by-nkn_k) variance of that feature:

P(xj∣y=k)=12πσkj2exp⁡ ⁣(−(xj−μkj)22σkj2),μ^kj=1nk∑i:yi=kxij,σ^kj2=1nk∑i:yi=k(xij−μ^kj)2P(x_j\mid y=k)=\frac{1}{\sqrt{2\pi\sigma_{kj}^2}}\exp\!\Big(-\frac{(x_j-\mu_{kj})^2}{2\sigma_{kj}^2}\Big),\quad \hat\mu_{kj}=\frac{1}{n_k}\sum_{i:y_i=k}x_{ij},\quad \hat\sigma^2_{kj}=\frac{1}{n_k}\sum_{i:y_i=k}(x_{ij}-\hat\mu_{kj})^2
μkj, σkj2\mu_{kj},\ \sigma^2_{kj}
mean and variance of feature jj within class kk
xijx_{ij}
feature jj of training row ii

Multiplying dd one-dimensional Gaussians gives a dd-dimensional Gaussian with a diagonal covariance: its contours are ellipses aligned with the axes. So Gaussian NB is exactly quadratic discriminant analysis with every off-diagonal covariance forced to zero; tie the variances across classes and it becomes a diagonal version of linear discriminant analysis (see those two articles for the full-covariance story). scikit-learn's "var_smoothing" adds ϵ⋅max⁡jVar⁡(xj)\epsilon\cdot\max_j\operatorname{Var}(x_j) to every σkj2\sigma^2_{kj} (default ϵ=10−9\epsilon=10^{-9}) so a constant feature in some class does not produce a zero variance and an infinite density.

Naive Bayes is a linear classifier (for discrete features). For two classes, the log-odds is a sum of per-feature terms: log⁡P(y=1∣x)P(y=0∣x)=log⁡π1π0+∑jlog⁡P(xj∣1)P(xj∣0)\log\frac{P(y=1\mid x)}{P(y=0\mid x)}=\log\frac{\pi_1}{\pi_0}+\sum_j\log\frac{P(x_j\mid 1)}{P(x_j\mid 0)}. With one-hot or count features each term is linear in the features, so the decision boundary is a hyperplane — the same functional form as logistic regression.

Worked example: play tennis, by hand

The same 14 days used in the decision-trees article: four categorical features and whether tennis was played (9 Yes, 5 No). We classify a new day: Sunny, Cool, High humidity, Strong wind.

DayOutlookTempHumidityWindPlay
1SunnyHotHighWeakNo
2SunnyHotHighStrongNo
3OvercastHotHighWeakYes
4RainMildHighWeakYes
5RainCoolNormalWeakYes
6RainCoolNormalStrongNo
7OvercastCoolNormalStrongYes
8SunnyMildHighWeakNo
9SunnyCoolNormalWeakYes
10RainMildNormalWeakYes
11SunnyMildNormalStrongYes
12OvercastMildHighStrongYes
13OvercastHotNormalWeakYes
14RainMildHighStrongNo
Play-tennis data (Quinlan, 1986). Highlighted: the 5 No days, from which every No likelihood is counted.

Training is one pass that fills the count table below. That is the entire model.

Feature = valueYes countNo countP(value | Yes)P(value | No)
Outlook = Sunny232/9 = 0.2223/5 = 0.600
Outlook = Overcast404/9 = 0.4440/5 = 0
Outlook = Rain323/9 = 0.3332/5 = 0.400
Temp = Hot222/9 = 0.2222/5 = 0.400
Temp = Mild424/9 = 0.4442/5 = 0.400
Temp = Cool313/9 = 0.3331/5 = 0.200
Humidity = High343/9 = 0.3334/5 = 0.800
Humidity = Normal616/9 = 0.6671/5 = 0.200
Wind = Weak626/9 = 0.6672/5 = 0.400
Wind = Strong333/9 = 0.3333/5 = 0.600
The fitted model: counts per class (Yes out of 9, No out of 5). Highlighted: the four values of the new day.

Classify (Sunny, Cool, High, Strong) with raw counts (α = 0)

  1. 1Priors: P(Yes)=9/14=0.643P(\text{Yes})=9/14=0.643, P(No)=5/14=0.357P(\text{No})=5/14=0.357.
  2. 2Yes score: 914⋅29⋅39⋅39⋅39=0.643⋅0.222⋅0.333⋅0.333⋅0.333=0.005291\tfrac{9}{14}\cdot\tfrac29\cdot\tfrac39\cdot\tfrac39\cdot\tfrac39=0.643\cdot0.222\cdot0.333\cdot0.333\cdot0.333=0.005291.
  3. 3No score: 514⋅35⋅15⋅45⋅35=0.357⋅0.600⋅0.200⋅0.800⋅0.600=0.020571\tfrac{5}{14}\cdot\tfrac35\cdot\tfrac15\cdot\tfrac45\cdot\tfrac35=0.357\cdot0.600\cdot0.200\cdot0.800\cdot0.600=0.020571.
  4. 4Normalise: P(No∣x)=0.0205710.005291+0.020571=0.795P(\text{No}\mid x)=\dfrac{0.020571}{0.005291+0.020571}=0.795, P(Yes∣x)=0.205P(\text{Yes}\mid x)=0.205.
  5. 5Predict No. The prior favoured Yes, but Sunny, High humidity and Strong wind are all much more typical of No days; only Cool argues for Yes.

The same computation in log-odds makes the "vote" of each feature explicit. Each row is log⁡P(⋅∣Yes)−log⁡P(⋅∣No)\log P(\cdot\mid\text{Yes})-\log P(\cdot\mid\text{No}); positive favours Yes.

TermYesNolog ratio (Yes : No)
prior9/145/14+0.588
Outlook = Sunny2/93/5−0.993
Temp = Cool3/91/5+0.511
Humidity = High3/94/5−0.875
Wind = Strong3/93/5−0.588
total log-odds−1.358
The evidence ledger for the new day. The sum −1.358 gives P(Yes∣x)=σ(−1.358)=0.205P(\text{Yes}\mid x)=\sigma(-1.358)=0.205, the same answer as above.

The same day with Laplace smoothing (α = 1)

  1. 1Each Yes likelihood becomes (c+1)/(9+Vj)(c+1)/(9+V_j) and each No likelihood (c+1)/(5+Vj)(c+1)/(5+V_j), with Vj=3,3,2,2V_j=3,3,2,2 values for Outlook, Temp, Humidity, Wind.
  2. 2Yes: 914⋅312⋅412⋅411⋅411=0.007084\tfrac{9}{14}\cdot\tfrac{3}{12}\cdot\tfrac{4}{12}\cdot\tfrac{4}{11}\cdot\tfrac{4}{11}=0.007084. No: 514⋅48⋅28⋅57⋅47=0.018222\tfrac{5}{14}\cdot\tfrac{4}{8}\cdot\tfrac{2}{8}\cdot\tfrac{5}{7}\cdot\tfrac{4}{7}=0.018222.
  3. 3P(No∣x)=0.018222/(0.007084+0.018222)=0.720P(\text{No}\mid x)=0.018222/(0.007084+0.018222)=0.720. Still No, but less confident: smoothing pulled every ratio toward 1, and the small No class (5 rows) is pulled hardest.

On its own 14 training days the model (with either α) gets 13 right; it misses day 6 (Rain, Cool, Normal, Strong → No), where the Cool and Normal votes for Yes outweigh Rain and Strong.

▸ naive bayes
Outlook
Temp
Humidity
Wind
α
evidence ledger · log P(·|Yes) − log P(·|No), in nats← favours Nofavours Yes →-4-3-2-10+1+2+3+4priorYes 9/14 · No 5/14+0.588Outlook = SunnyYes 2/9=0.222 · No 3/5=0.600−0.993Temp = CoolYes 3/9=0.333 · No 1/5=0.200+0.511Humidity = HighYes 3/9=0.333 · No 4/5=0.800−0.875Wind = StrongYes 3/9=0.333 · No 3/5=0.600−0.588log-odds (Yes : No)−1.358posteriorP(Yes | x) = 20.5%P(No | x) = 79.5%
counts from the 14 days (Yes/No) — training is just this table
OutlookSunny 2/3 · Overcast 4/0 · Rain 3/2
TempHot 2/2 · Mild 4/2 · Cool 3/1
HumidityHigh 3/4 · Normal 6/1
WindWeak 6/2 · Strong 3/3
P(x, Yes) = 0.00529 · P(x, No) = 0.02057 → predict No
fig.Two explorers. Counting · play tennis: pick the new day's four values and the smoothing α; each bar is one feature's log-likelihood ratio added to the running log-odds, so you can watch the evidence accumulate. The default (Sunny, Cool, High, Strong, α = 0) ends at −1.358 → P(Yes) = 20.5%; α = 1 gives 28.0%; choosing Overcast with α = 0 produces the +∞ zero-count bar. Gaussian · 2-D: 16 points; each class gets an axis-aligned Gaussian per feature (curves on the margins, 1σ/2σ ellipses in the plane). Drag the query: at (4.6, 5.4) per-class variances give P(B) = 0.753, but switching to shared variances flips it to P(A) = 0.813 and the curved boundary becomes a straight line.

The algorithm, step by step

fit(X, y) and predict(x)

  1. 1Priors. For each class kk: log⁡πk=log⁡(nk/n)\log\pi_k=\log(n_k/n) (or use priors you supply, e.g. known base rates).
  2. 2Likelihood tables. For each class and feature, compute sufficient statistics in one pass: value counts (categorical), token counts (multinomial), presence counts (Bernoulli), or mean and variance (Gaussian).
  3. 3Smooth and log. Turn counts into log⁡c+αnk+αV\log\frac{c+\alpha}{n_k+\alpha V} (or add "var_smoothing" to variances) and store the log-probabilities.
  4. 4Score. For a new xx: sk=log⁡πk+∑jlog⁡P(xj∣y=k)s_k=\log\pi_k+\sum_j\log P(x_j\mid y=k) — a table lookup and an addition per feature.
  5. 5Normalise. P(y=k∣x)=exp⁡(sk−m)/∑k′exp⁡(sk′−m)P(y=k\mid x)=\exp(s_k-m)/\sum_{k'}\exp(s_{k'}-m) with m=max⁡kskm=\max_k s_k; predict arg⁡max⁡ksk\arg\max_k s_k.

Cost

Training time
O(n d)O(n\,d) — one pass; trivially parallel and incremental (add new rows' counts with "partial_fit")
Prediction time
O(d K)O(d\,K) per example; for sparse text only the non-zero words cost anything
Memory
O(K∑jVj)O(K\sum_j V_j) for count tables, O(2Kd)O(2Kd) for Gaussians — independent of nn

Variants and hyperparameters

The variants differ only in the event model P(xj∣y)P(x_j\mid y); the prior and the arg-max are the same.

VariantFeature typeP(x_j | y=k)Typical use
CategoricalNBcategories(ckjv+α)/(nk+αVj)(c_{kjv}+\alpha)/(n_k+\alpha V_j)tabular categorical data
GaussianNBreal numbersN(xj;μkj,σkj2)\mathcal N(x_j;\mu_{kj},\sigma^2_{kj})sensor readings, quick baselines
MultinomialNBcounts (word frequencies)θkj xj\theta_{kj}^{\,x_j}, θkj=Nkj+αNk+α∣V∣\theta_{kj}=\frac{N_{kj}+\alpha}{N_k+\alpha|V|}text classification, spam
BernoulliNBbinary (word present?)θkjxj(1−θkj)1−xj\theta_{kj}^{x_j}(1-\theta_{kj})^{1-x_j}short texts; absence counts as evidence
ComplementNBcountsmultinomial estimated from all other classesimbalanced text classes
The naive Bayes family in scikit-learn.

Multinomial NB on a tiny spam corpus. Three spam messages ("free money now", "free free prize", "win money free") and three ham ("meeting at noon", "money for lunch", "noon meeting moved"). Each class has Nk=9N_k=9 tokens and the vocabulary has ∣V∣=11|V|=11 words. With α=1\alpha=1, P(w∣k)=(Nkw+1)/20P(w\mid k)=(N_{kw}+1)/20:

Wordcount in spamcount in hamP(w | spam)P(w | ham)
free405/20 = 0.251/20 = 0.05
money213/20 = 0.152/20 = 0.10
meeting021/20 = 0.053/20 = 0.15
Classifying "free money meeting". Spam: 0.5⋅0.25⋅0.15⋅0.05=0.00093750.5\cdot0.25\cdot0.15\cdot0.05=0.0009375; ham: 0.5⋅0.05⋅0.10⋅0.15=0.0003750.5\cdot0.05\cdot0.10\cdot0.15=0.000375 → P(spam) = 0.714.

Note that the multinomial model ignores the 8 vocabulary words absent from the message; Bernoulli NB multiplies in 1−θ1-\theta for each of them, and on this corpus gives P(spam) = 0.842 instead.

VariancesClass A: μ, σ²Class B: μ, σ²log score Alog score BPosterior
per class(3.06, 6.21), (0.35, 0.28)(6.50, 4.04), (1.63, 1.13)−0.693 − 3.732 − 1.453 = −5.878−0.693 − 2.270 − 1.800 = −4.763P(B) = 0.753
shared(3.06, 6.21), (0.99, 0.71)(6.50, 4.04), (0.99, 0.71)−4.011−5.481P(A) = 0.813
Gaussian NB on the figure's 16 points, query (4.6, 5.4). Log score = log prior + log N(x₁) + log N(x₂). Sharing the variances changes the answer.

With per-class variances, the wide class B has a density that decays slowly, so it claims the space between the classes and the boundary curves around the tight class A. With a shared variance the quadratic terms cancel in the log-ratio and the boundary is a straight line through the midpoint of the two means (the priors are equal here).

What to tune

alpha
(Categorical/Multinomial/Bernoulli) Larger → flatter likelihoods, softer posteriors. Grid-search on a log scale, e.g. 10−3…1010^{-3}\ldots10; text classifiers often prefer α<1\alpha<1.
var_smoothing
(Gaussian) The ϵ\epsilon in σ2+ϵmax⁡Var⁡\sigma^2+\epsilon\max\operatorname{Var}. Larger values widen every Gaussian, which smooths the boundary toward linear. Search 10−11…10−110^{-11}\ldots10^{-1}.
priors / fit_prior
Override πk\pi_k when the deployment base rates differ from the training mix; "fit_prior=False" uses uniform priors.
event model
Usually the biggest decision: counts → Multinomial/Complement, binary → Bernoulli, real → Gaussian (after a transform that makes features roughly normal), categories → Categorical. Mixed types: fit one model per block and add their log-likelihoods.
features
Feature selection matters more than for most models: removing near-duplicate features reduces double counting.

In code

python
1import numpy as np
2
3class CategoricalNB:
4 """Naive Bayes for categorical features, with Laplace/Lidstone smoothing alpha."""
5 def __init__(self, alpha=1.0):
6 self.alpha = alpha
7
8 def fit(self, X, y):
9 X, y = np.asarray(X), np.asarray(y)
10 self.classes_ = np.unique(y)
11 self.values_ = [np.unique(X[:, j]) for j in range(X.shape[1])]
12 self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])
13 # log_lik_[j][k, v] = log P(x_j = value v | class k)
14 self.log_lik_ = []
15 for j, vals in enumerate(self.values_):
16 counts = np.array([[np.sum(X[y == c, j] == v) for v in vals] for c in self.classes_], float)
17 probs = (counts + self.alpha) / (counts.sum(1, keepdims=True) + self.alpha * len(vals))
18 with np.errstate(divide="ignore"): # alpha=0 gives log(0) = -inf
19 self.log_lik_.append(np.log(probs))
20 return self
21
22 def joint_log(self, x):
23 """log P(y=k) + sum_j log P(x_j | y=k), for every class k."""
24 s = self.log_prior_.copy()
25 for j, v in enumerate(x):
26 s += self.log_lik_[j][:, np.searchsorted(self.values_[j], v)]
27 return s
28
29 def predict_proba(self, X):
30 S = np.array([self.joint_log(x) for x in np.asarray(X)])
31 S -= S.max(1, keepdims=True) # log-sum-exp trick
32 P = np.exp(S)
33 return P / P.sum(1, keepdims=True)
34
35 def predict(self, X):
36 return self.classes_[self.predict_proba(X).argmax(1)]
37
38class GaussianNB:
39 def fit(self, X, y, eps=1e-9):
40 self.classes_ = np.unique(y)
41 self.mu_ = np.array([X[y == c].mean(0) for c in self.classes_])
42 self.var_ = np.array([X[y == c].var(0) for c in self.classes_]) + eps * X.var(0).max()
43 self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])
44 return self
45
46 def predict_proba(self, X):
47 S = self.log_prior_ - 0.5 * (np.log(2 * np.pi * self.var_)[None]
48 + (X[:, None, :] - self.mu_[None]) ** 2 / self.var_[None]).sum(-1)
49 P = np.exp(S - S.max(1, keepdims=True))
50 return P / P.sum(1, keepdims=True)
51
52D = np.array("""
53Sunny Hot High Weak No Sunny Hot High Strong No Overcast Hot High Weak Yes
54Rain Mild High Weak Yes Rain Cool Normal Weak Yes Rain Cool Normal Strong No
55Overcast Cool Normal Strong Yes Sunny Mild High Weak No Sunny Cool Normal Weak Yes
56Rain Mild Normal Weak Yes Sunny Mild Normal Strong Yes Overcast Mild High Strong Yes
57Overcast Hot Normal Weak Yes Rain Mild High Strong No""".split()).reshape(-1, 5)
58X, y = D[:, :4], D[:, 4]
59
60query = [["Sunny", "Cool", "High", "Strong"]]
61for a in (0.0, 1.0):
62 nb = CategoricalNB(alpha=a).fit(X, y)
63 print(f"alpha={a}: classes {nb.classes_}, P = {nb.predict_proba(query).round(4)}, "
64 f"predict {nb.predict(query)}, train acc {np.mean(nb.predict(X) == y):.3f}")
65
66Xg = np.array([(2.2,6.1),(2.9,5.4),(3.4,6.8),(3.8,5.9),(2.6,7.0),(3.1,6.3),(4.0,6.6),(2.5,5.6),
67 (5.0,3.2),(6.1,5.0),(7.8,3.6),(8.6,4.9),(6.8,2.4),(4.6,4.6),(7.2,5.6),(5.9,3.0)])
68yg = np.repeat([0, 1], 8)
69g = GaussianNB().fit(Xg, yg)
70print("means", g.mu_.round(3).tolist(), "variances", g.var_.round(3).tolist())
71print("P(class | x=(4.6, 5.4)) =", g.predict_proba(np.array([[4.6, 5.4]])).round(3))
From scratch with NumPy: categorical NB with smoothing, and Gaussian NB. Prints P = [[0.7954 0.2046]] for α = 0, [[0.7201 0.2799]] for α = 1, training accuracy 0.929, and P = [[0.247 0.753]] for the 2-D query.
python
1import pandas as pd
2from sklearn.naive_bayes import CategoricalNB, GaussianNB, MultinomialNB
3from sklearn.preprocessing import OrdinalEncoder
4from sklearn.feature_extraction.text import CountVectorizer
5
6df = pd.DataFrame({
7 "Outlook": "Sunny Sunny Overcast Rain Rain Rain Overcast Sunny Sunny Rain Sunny Overcast Overcast Rain".split(),
8 "Temp": "Hot Hot Hot Mild Cool Cool Cool Mild Cool Mild Mild Mild Hot Mild".split(),
9 "Humidity": "High High High High Normal Normal Normal High Normal Normal Normal High Normal High".split(),
10 "Wind": "Weak Strong Weak Weak Weak Strong Strong Weak Weak Weak Strong Strong Weak Strong".split(),
11 "Play": "No No Yes Yes Yes No Yes No Yes Yes Yes Yes Yes No".split(),
12})
13enc = OrdinalEncoder()
14X = enc.fit_transform(df.drop(columns="Play"))
15nb = CategoricalNB(alpha=1.0).fit(X, df.Play)
16
17new = pd.DataFrame([["Sunny", "Cool", "High", "Strong"]], columns=df.columns[:4])
18print(nb.classes_, nb.predict_proba(enc.transform(new)).round(4)) # ['No' 'Yes'] [[0.7201 0.2799]]
19print("train accuracy:", round(nb.score(X, df.Play), 3)) # 0.929 (day 6 is missed)
20
21# Text: multinomial NB on word counts
22docs = ["free money now", "free free prize", "win money free",
23 "meeting at noon", "money for lunch", "noon meeting moved"]
24spam = [1, 1, 1, 0, 0, 0]
25cv = CountVectorizer()
26mnb = MultinomialNB(alpha=1.0).fit(cv.fit_transform(docs), spam)
27print(mnb.predict_proba(cv.transform(["free money meeting"])).round(4)) # [[0.2857 0.7143]]
28
29# Continuous features: GaussianNB (var_smoothing adds a small fraction of the largest variance)
30Xg = [(2.2,6.1),(2.9,5.4),(3.4,6.8),(3.8,5.9),(2.6,7.0),(3.1,6.3),(4.0,6.6),(2.5,5.6),
31 (5.0,3.2),(6.1,5.0),(7.8,3.6),(8.6,4.9),(6.8,2.4),(4.6,4.6),(7.2,5.6),(5.9,3.0)]
32g = GaussianNB().fit(Xg, [0] * 8 + [1] * 8)
33print(g.predict_proba([[4.6, 5.4]]).round(3)) # [[0.247 0.753]]
The same three examples with scikit-learn. Outputs are shown in the comments.

Use when

  • +You have little labelled data, or very many features (text, bag-of-words, genomics): few parameters per feature means low variance.
  • +You need training and prediction to be extremely fast, incremental, or done on streaming data.
  • +You want a strong, explainable baseline: every prediction decomposes into per-feature log-odds.
  • +Classes may be added later — each class's model is fitted independently.

Avoid when

  • −Features are strongly correlated or redundant: evidence is double-counted and probabilities become overconfident.
  • −You need calibrated probabilities (risk scores, expected cost) without a calibration step.
  • −Interactions matter (XOR-like patterns): the model cannot represent them at all.
  • −Data is plentiful: logistic regression, gradient boosting or a neural net will usually be more accurate.

Where it's used

Spam filtering

The classic application (Sahami et al. 1998; Paul Graham's 'A Plan for Spam'): multinomial or Bernoulli NB over words, updated incrementally as users flag messages.

Text and document categorisation

News topics, support-ticket routing, language identification — MultinomialNB/ComplementNB on TF or TF-IDF features is a hard-to-beat fast baseline.

Sentiment analysis

Positive/negative reviews; NB features combined with an SVM (NBSVM) were a strong pre-transformer baseline.

Medical and fault diagnosis

Symptoms → disease or alarm codes → fault, where each piece of evidence contributes an interpretable likelihood ratio.

Check yourself

With α=0\alpha=0, a test email contains a word that never appeared in any training spam. What is its posterior probability of spam?

Check yourself

In the spam corpus above (spam has 9 tokens, the vocabulary has 11 words, "free" appears 4 times in spam), what is P(free∣spam)P(\text{free}\mid\text{spam}) with α=1\alpha=1?

Check yourself

You duplicate one feature column three times before training naive Bayes. What changes?

Check yourself

Gaussian naive Bayes with the same variance for each feature in every class has a decision boundary that is…

Remember

  • ✓Naive Bayes applies Bayes' rule with the assumption that features are independent given the class: P(y∣x)∝P(y)∏jP(xj∣y)P(y\mid x)\propto P(y)\prod_j P(x_j\mid y).
  • ✓Training is closed-form maximum likelihood: count tables (categorical, multinomial, Bernoulli) or per-class means and variances (Gaussian) — one pass, O(nd).
  • ✓Work in log space: each feature adds a log-likelihood ratio to the prior log-odds, which makes predictions explainable and avoids underflow.
  • ✓Smooth counts with α\alpha (Laplace = 1): otherwise one unseen value forces a posterior of exactly 0.
  • ✓Choose the event model to match the features; Gaussian NB is QDA with diagonal covariances, and discrete NB is a linear classifier like logistic regression.
  • ✓It ranks classes well even when independence is false, but its probabilities are overconfident — calibrate them before trusting them.
← back to ML Algorithms roadmap