# Naive Bayes

> Assume features are independent given the class, so P(y|x) ∝ P(y) Πᵢ P(xᵢ|y) — training is just counting.

Source: https://www.mlfoundry.in/learn/naive-bayes  
Section: ML Algorithms › Supervised · Classification  
Level: core · 17 min read · updated 2026-10-07

**Naive Bayes:** is a family of probabilistic classifiers that model how each class *generates* data — a prior $P(y)$ and, for every feature separately, a distribution $P(x_j\mid y)$ — and then use Bayes' rule to turn that model around into $P(y\mid x)$. The "naive" part is the assumption that the features are **conditionally independent given the class**, which lets the joint likelihood factor into a product of one-feature pieces that can each be estimated by counting.

- **Task:** Classification (binary or multi-class), with class probabilities
- **Input → output:** feature vector $x$ → posterior $P(y=k\mid x)$ for every class; predict the largest
- **Learns:** class priors $P(y=k)$ and one small distribution per (feature, class): a count table, a Gaussian $(\mu,\sigma^2)$, or word frequencies
- **Training:** one pass over the data: $O(n\,d)$ — no optimisation, no iterations
- **Prediction:** $O(d\,K)$: add $d$ log-likelihoods for each of $K$ classes
- **Hyperparameters:** smoothing $\alpha$ (counts), "var_smoothing" (Gaussian), class priors, and the choice of per-feature distribution
- **Assumes:** features independent given the class; each feature follows the chosen family (categorical, Gaussian, multinomial, Bernoulli)

## The problem

Bayes' rule tells us exactly how to classify: pick the class with the largest $P(y\mid x)\propto P(y)\,P(x\mid y)$. The trouble is $P(x\mid y)$ — the probability of an *entire feature vector*. With just 30 yes/no features there are $2^{30}\approx 10^9$ possible vectors per class; we would need billions of examples to estimate their probabilities, and almost every new input would be a vector we have never seen, with estimated probability 0.

Discriminative models such as **logistic regression** dodge the problem by modelling $P(y\mid x)$ directly, but they need iterative optimisation and enough data to pin down a weight per feature. Naive Bayes takes the opposite route: keep the generative story, but make one bold simplification so that $P(x\mid y)$ needs only $d$ one-dimensional estimates instead of one $d$-dimensional one.

## The idea

Pretend that, *once you know the class*, the features tell you nothing about each other. Knowing an email is spam, whether it contains "free" says nothing extra about whether it contains "money". Then the probability of the whole vector is just the product of the individual feature probabilities, and each of those is estimated from the training rows of that class — usually by counting.

To classify, every class starts with its prior and then each feature **votes**: its likelihood under that class multiplies in. In log space the votes add up, so naive Bayes is an evidence ledger — each feature contributes a fixed amount of log-odds for or against each class, and the class with the largest total wins.

**Vocabulary**

- **Prior:** $P(y=k)$: how common class $k$ is before looking at the features — the fraction of training rows with label $k$
- **Likelihood:** $P(x_j\mid y=k)$: how probable feature value $x_j$ is among examples of class $k$
- **Posterior:** $P(y=k\mid x)$: the class probability after seeing the features; what we predict from
- **Evidence:** $P(x)=\sum_k P(y=k)P(x\mid y=k)$: the normaliser that makes posteriors sum to 1; it is the same for every class, so the arg-max ignores it
- **Conditional independence:** $P(x_i,x_j\mid y)=P(x_i\mid y)P(x_j\mid y)$: independence *within each class*. Features can still be strongly correlated overall (through the class)
- **Generative model:** one that models $P(x,y)=P(y)P(x\mid y)$ — how data is produced — rather than only the boundary $P(y\mid x)$
- **Smoothing (α):** pseudo-counts added to every count so that an unseen value gets a small probability instead of zero
- **Event model:** the per-feature distribution family: categorical, Gaussian, multinomial (word counts) or Bernoulli (word present/absent)

## The math

Start from Bayes' rule for a class $k$ and a feature vector $x=(x_1,\dots,x_d)$:

$$
P(y=k\mid x)=\frac{P(y=k)\,P(x\mid y=k)}{P(x)}
$$

- $P(y=k)$: prior of class $k$
- $P(x\mid y=k)$: class-conditional likelihood of the whole vector $x$
- $P(x)$: evidence $\sum_{k'}P(y=k')P(x\mid y=k')$, identical for all classes

The chain rule gives $P(x\mid y)=P(x_1\mid y)\,P(x_2\mid y,x_1)\cdots P(x_d\mid y,x_1,\dots,x_{d-1})$ exactly. The naive assumption drops every conditioning variable except the class, $P(x_j\mid y,x_1,\dots,x_{j-1})\approx P(x_j\mid y)$, so

$$
P(y=k\mid x)\;\propto\;\pi_k\prod_{j=1}^{d}P(x_j\mid y=k),\qquad \hat y=\arg\max_k\Big[\log\pi_k+\sum_{j=1}^{d}\log P(x_j\mid y=k)\Big]
$$

- $\pi_k$: prior $P(y=k)$, estimated as $n_k/n$
- $d$: number of features
- $\hat y$: predicted class (the maximum a posteriori class)

We take logs for two reasons: the product of hundreds of small probabilities underflows to 0 in floating point, and sums are cheaper and easier to read than products. Posteriors are recovered with the log-sum-exp trick: subtract the largest class score, exponentiate, normalise.

## Training is counting: the maximum-likelihood estimate

For a categorical feature $j$ with values $v=1,\dots,V_j$, the model for class $k$ is a probability vector $\theta_{kjv}=P(x_j=v\mid y=k)$. Let $c_{kjv}$ be the number of class-$k$ rows whose feature $j$ equals $v$. The log-likelihood of the training data that involves this vector is $\sum_v c_{kjv}\log\theta_{kjv}$, to be maximised subject to $\sum_v\theta_{kjv}=1$. With a Lagrange multiplier $\lambda$:

$$
\frac{\partial}{\partial\theta_{kjv}}\Big[\sum_{v}c_{kjv}\log\theta_{kjv}-\lambda\Big(\sum_v\theta_{kjv}-1\Big)\Big]=\frac{c_{kjv}}{\theta_{kjv}}-\lambda=0\;\Rightarrow\;\hat\theta_{kjv}=\frac{c_{kjv}}{n_k}
$$

- $c_{kjv}$: count of class-$k$ rows with $x_j=v$
- $n_k$: number of class-$k$ rows; the constraint forces $\lambda=\sum_v c_{kjv}=n_k$
- $\hat\theta_{kjv}$: estimated $P(x_j=v\mid y=k)$

So the maximum-likelihood fit is literally a frequency table, and because the log-likelihood splits into one independent term per (class, feature), every table is fitted on its own in a single pass. The same argument gives $\hat\pi_k=n_k/n$.

**Smoothing.** If a value never occurs with class $k$, $\hat\theta_{kjv}=0$ and *any* input with that value gets posterior 0 for class $k$, no matter how strongly every other feature points to it. The fix adds $\alpha$ pseudo-counts to every cell:

$$
\hat\theta_{kjv}=\frac{c_{kjv}+\alpha}{n_k+\alpha V_j}
$$

- $\alpha$: smoothing strength: $\alpha=1$ is Laplace smoothing, $0<\alpha<1$ Lidstone; $\alpha=0$ is the raw MLE
- $V_j$: number of distinct values of feature $j$ (so the probabilities still sum to 1)

This is the posterior mean of $\theta$ under a symmetric Dirichlet($\alpha$) prior: $\alpha$ imaginary observations of each value. Mathematically $\alpha$ shrinks every estimate toward the uniform $1/V_j$, and the shrinkage weight is $\alpha V_j/(n_k+\alpha V_j)$ — large when the class is rare, negligible when $n_k\gg\alpha V_j$. Larger $\alpha$ therefore means flatter likelihood ratios, less extreme posteriors, and more bias; smaller $\alpha$ means more variance.

## Continuous features: Gaussian naive Bayes

For a real-valued feature, the usual choice is one Gaussian per (class, feature). The maximum-likelihood estimates are again closed-form — the class's sample mean and (biased, divide-by-$n_k$) variance of that feature:

$$
P(x_j\mid y=k)=\frac{1}{\sqrt{2\pi\sigma_{kj}^2}}\exp\!\Big(-\frac{(x_j-\mu_{kj})^2}{2\sigma_{kj}^2}\Big),\quad \hat\mu_{kj}=\frac{1}{n_k}\sum_{i:y_i=k}x_{ij},\quad \hat\sigma^2_{kj}=\frac{1}{n_k}\sum_{i:y_i=k}(x_{ij}-\hat\mu_{kj})^2
$$

- $\mu_{kj},\ \sigma^2_{kj}$: mean and variance of feature $j$ within class $k$
- $x_{ij}$: feature $j$ of training row $i$

Multiplying $d$ one-dimensional Gaussians gives a $d$-dimensional Gaussian with a **diagonal** covariance: its contours are ellipses aligned with the axes. So Gaussian NB is exactly **quadratic discriminant analysis** with every off-diagonal covariance forced to zero; tie the variances across classes and it becomes a diagonal version of **linear discriminant analysis** (see those two articles for the full-covariance story). scikit-learn's "var_smoothing" adds $\epsilon\cdot\max_j\operatorname{Var}(x_j)$ to every $\sigma^2_{kj}$ (default $\epsilon=10^{-9}$) so a constant feature in some class does not produce a zero variance and an infinite density.

**Naive Bayes is a linear classifier (for discrete features).** For two classes, the log-odds is a sum of per-feature terms: $\log\frac{P(y=1\mid x)}{P(y=0\mid x)}=\log\frac{\pi_1}{\pi_0}+\sum_j\log\frac{P(x_j\mid 1)}{P(x_j\mid 0)}$. With one-hot or count features each term is linear in the features, so the decision boundary is a hyperplane — the same functional form as logistic regression.

> **A generative–discriminative pair:** Categorical naive Bayes and logistic regression on one-hot features describe the *same* family of posteriors $\sigma(w^\top x+b)$. They differ only in how $w$ is chosen: naive Bayes sets each weight from its own feature's counts (a closed form, assuming independence); logistic regression fits all weights jointly to maximise $P(y\mid x)$. Ng and Jordan (2002) showed the consequence: naive Bayes approaches its (higher) asymptotic error with far fewer examples, while logistic regression wins once data is plentiful — because it can correct for correlated features that naive Bayes double-counts.

## Worked example: play tennis, by hand

The same 14 days used in the decision-trees article: four categorical features and whether tennis was played (9 Yes, 5 No). We classify a new day: **Sunny, Cool, High humidity, Strong wind**.

Play-tennis data (Quinlan, 1986). Highlighted: the 5 No days, from which every No likelihood is counted.

| Day | Outlook | Temp | Humidity | Wind | Play |
| --- | --- | --- | --- | --- | --- |
| 1 | Sunny | Hot | High | Weak | No |
| 2 | Sunny | Hot | High | Strong | No |
| 3 | Overcast | Hot | High | Weak | Yes |
| 4 | Rain | Mild | High | Weak | Yes |
| 5 | Rain | Cool | Normal | Weak | Yes |
| 6 | Rain | Cool | Normal | Strong | No |
| 7 | Overcast | Cool | Normal | Strong | Yes |
| 8 | Sunny | Mild | High | Weak | No |
| 9 | Sunny | Cool | Normal | Weak | Yes |
| 10 | Rain | Mild | Normal | Weak | Yes |
| 11 | Sunny | Mild | Normal | Strong | Yes |
| 12 | Overcast | Mild | High | Strong | Yes |
| 13 | Overcast | Hot | Normal | Weak | Yes |
| 14 | Rain | Mild | High | Strong | No |

**Training** is one pass that fills the count table below. That is the entire model.

The fitted model: counts per class (Yes out of 9, No out of 5). Highlighted: the four values of the new day.

| Feature = value | Yes count | No count | P(value \| Yes) | P(value \| No) |
| --- | --- | --- | --- | --- |
| **Outlook = Sunny** | 2 | 3 | 2/9 = 0.222 | 3/5 = 0.600 |
| Outlook = Overcast | 4 | 0 | 4/9 = 0.444 | 0/5 = **0** |
| Outlook = Rain | 3 | 2 | 3/9 = 0.333 | 2/5 = 0.400 |
| Temp = Hot | 2 | 2 | 2/9 = 0.222 | 2/5 = 0.400 |
| Temp = Mild | 4 | 2 | 4/9 = 0.444 | 2/5 = 0.400 |
| **Temp = Cool** | 3 | 1 | 3/9 = 0.333 | 1/5 = 0.200 |
| **Humidity = High** | 3 | 4 | 3/9 = 0.333 | 4/5 = 0.800 |
| Humidity = Normal | 6 | 1 | 6/9 = 0.667 | 1/5 = 0.200 |
| Wind = Weak | 6 | 2 | 6/9 = 0.667 | 2/5 = 0.400 |
| **Wind = Strong** | 3 | 3 | 3/9 = 0.333 | 3/5 = 0.600 |

**Classify (Sunny, Cool, High, Strong) with raw counts (α = 0)**

1. **Priors:** $P(\text{Yes})=9/14=0.643$, $P(\text{No})=5/14=0.357$.
2. **Yes score:** $\tfrac{9}{14}\cdot\tfrac29\cdot\tfrac39\cdot\tfrac39\cdot\tfrac39=0.643\cdot0.222\cdot0.333\cdot0.333\cdot0.333=0.005291$.
3. **No score:** $\tfrac{5}{14}\cdot\tfrac35\cdot\tfrac15\cdot\tfrac45\cdot\tfrac35=0.357\cdot0.600\cdot0.200\cdot0.800\cdot0.600=0.020571$.
4. **Normalise:** $P(\text{No}\mid x)=\dfrac{0.020571}{0.005291+0.020571}=0.795$, $P(\text{Yes}\mid x)=0.205$.
5. **Predict No.** The prior favoured Yes, but Sunny, High humidity and Strong wind are all much more typical of No days; only Cool argues for Yes.

The same computation in log-odds makes the "vote" of each feature explicit. Each row is $\log P(\cdot\mid\text{Yes})-\log P(\cdot\mid\text{No})$; positive favours Yes.

The evidence ledger for the new day. The sum −1.358 gives $P(\text{Yes}\mid x)=\sigma(-1.358)=0.205$, the same answer as above.

| Term | Yes | No | log ratio (Yes : No) |
| --- | --- | --- | --- |
| prior | 9/14 | 5/14 | +0.588 |
| Outlook = Sunny | 2/9 | 3/5 | −0.993 |
| Temp = Cool | 3/9 | 1/5 | +0.511 |
| Humidity = High | 3/9 | 4/5 | −0.875 |
| Wind = Strong | 3/9 | 3/5 | −0.588 |
| **total log-odds** |  |  | **−1.358** |

**The same day with Laplace smoothing (α = 1)**

1. Each Yes likelihood becomes $(c+1)/(9+V_j)$ and each No likelihood $(c+1)/(5+V_j)$, with $V_j=3,3,2,2$ values for Outlook, Temp, Humidity, Wind.
2. **Yes:** $\tfrac{9}{14}\cdot\tfrac{3}{12}\cdot\tfrac{4}{12}\cdot\tfrac{4}{11}\cdot\tfrac{4}{11}=0.007084$. **No:** $\tfrac{5}{14}\cdot\tfrac{4}{8}\cdot\tfrac{2}{8}\cdot\tfrac{5}{7}\cdot\tfrac{4}{7}=0.018222$.
3. $P(\text{No}\mid x)=0.018222/(0.007084+0.018222)=0.720$. Still **No**, but less confident: smoothing pulled every ratio toward 1, and the small No class (5 rows) is pulled hardest.

> **One zero wipes out all other evidence:** Change the day to **Overcast**, Cool, High, Strong. No Overcast day was ever a No, so with $\alpha=0$ the No score is $\tfrac{5}{14}\cdot 0\cdots=0$ and $P(\text{Yes}\mid x)=1.000$ — certainty, even though High humidity and Strong wind point the other way. With $\alpha=1$, $P(\text{Overcast}\mid\text{No})=1/8$ and the posterior becomes a sane $P(\text{Yes}\mid x)=0.722$. Always smooth.

On its own 14 training days the model (with either α) gets 13 right; it misses day 6 (Rain, Cool, Normal, Strong → No), where the Cool and Normal votes for Yes outweigh Rain and Strong.

> Figure: Two explorers. Counting · play tennis: pick the new day's four values and the smoothing α; each bar is one feature's log-likelihood ratio added to the running log-odds, so you can watch the evidence accumulate. The default (Sunny, Cool, High, Strong, α = 0) ends at −1.358 → P(Yes) = 20.5%; α = 1 gives 28.0%; choosing Overcast with α = 0 produces the +∞ zero-count bar. Gaussian · 2-D: 16 points; each class gets an axis-aligned Gaussian per feature (curves on the margins, 1σ/2σ ellipses in the plane). Drag the query: at (4.6, 5.4) per-class variances give P(B) = 0.753, but switching to shared variances flips it to P(A) = 0.813 and the curved boundary becomes a straight line.

## The algorithm, step by step

**fit(X, y) and predict(x)**

1. **Priors.** For each class $k$: $\log\pi_k=\log(n_k/n)$ (or use priors you supply, e.g. known base rates).
2. **Likelihood tables.** For each class and feature, compute sufficient statistics in one pass: value counts (categorical), token counts (multinomial), presence counts (Bernoulli), or mean and variance (Gaussian).
3. **Smooth and log.** Turn counts into $\log\frac{c+\alpha}{n_k+\alpha V}$ (or add "var_smoothing" to variances) and store the log-probabilities.
4. **Score.** For a new $x$: $s_k=\log\pi_k+\sum_j\log P(x_j\mid y=k)$ — a table lookup and an addition per feature.
5. **Normalise.** $P(y=k\mid x)=\exp(s_k-m)/\sum_{k'}\exp(s_{k'}-m)$ with $m=\max_k s_k$; predict $\arg\max_k s_k$.

**Cost**

- **Training time:** $O(n\,d)$ — one pass; trivially parallel and **incremental** (add new rows' counts with "partial_fit")
- **Prediction time:** $O(d\,K)$ per example; for sparse text only the non-zero words cost anything
- **Memory:** $O(K\sum_j V_j)$ for count tables, $O(2Kd)$ for Gaussians — independent of $n$

## Variants and hyperparameters

The variants differ only in the event model $P(x_j\mid y)$; the prior and the arg-max are the same.

The naive Bayes family in scikit-learn.

| Variant | Feature type | P(x_j \| y=k) | Typical use |
| --- | --- | --- | --- |
| CategoricalNB | categories | $(c_{kjv}+\alpha)/(n_k+\alpha V_j)$ | tabular categorical data |
| GaussianNB | real numbers | $\mathcal N(x_j;\mu_{kj},\sigma^2_{kj})$ | sensor readings, quick baselines |
| MultinomialNB | counts (word frequencies) | $\theta_{kj}^{\,x_j}$, $\theta_{kj}=\frac{N_{kj}+\alpha}{N_k+\alpha\|V\|}$ | text classification, spam |
| BernoulliNB | binary (word present?) | $\theta_{kj}^{x_j}(1-\theta_{kj})^{1-x_j}$ | short texts; absence counts as evidence |
| ComplementNB | counts | multinomial estimated from all *other* classes | imbalanced text classes |

**Multinomial NB on a tiny spam corpus.** Three spam messages ("free money now", "free free prize", "win money free") and three ham ("meeting at noon", "money for lunch", "noon meeting moved"). Each class has $N_k=9$ tokens and the vocabulary has $|V|=11$ words. With $\alpha=1$, $P(w\mid k)=(N_{kw}+1)/20$:

Classifying "free money meeting". Spam: $0.5\cdot0.25\cdot0.15\cdot0.05=0.0009375$; ham: $0.5\cdot0.05\cdot0.10\cdot0.15=0.000375$ → P(spam) = 0.714.

| Word | count in spam | count in ham | P(w \| spam) | P(w \| ham) |
| --- | --- | --- | --- | --- |
| free | 4 | 0 | 5/20 = 0.25 | 1/20 = 0.05 |
| money | 2 | 1 | 3/20 = 0.15 | 2/20 = 0.10 |
| meeting | 0 | 2 | 1/20 = 0.05 | 3/20 = 0.15 |

Note that the multinomial model ignores the 8 vocabulary words *absent* from the message; Bernoulli NB multiplies in $1-\theta$ for each of them, and on this corpus gives P(spam) = 0.842 instead.

Gaussian NB on the figure's 16 points, query (4.6, 5.4). Log score = log prior + log N(x₁) + log N(x₂). Sharing the variances changes the answer.

| Variances | Class A: μ, σ² | Class B: μ, σ² | log score A | log score B | Posterior |
| --- | --- | --- | --- | --- | --- |
| per class | (3.06, 6.21), (0.35, 0.28) | (6.50, 4.04), (1.63, 1.13) | −0.693 − 3.732 − 1.453 = −5.878 | −0.693 − 2.270 − 1.800 = −4.763 | P(B) = 0.753 |
| shared | (3.06, 6.21), (0.99, 0.71) | (6.50, 4.04), (0.99, 0.71) | −4.011 | −5.481 | P(A) = 0.813 |

With per-class variances, the wide class B has a density that decays slowly, so it claims the space *between* the classes and the boundary curves around the tight class A. With a shared variance the quadratic terms cancel in the log-ratio and the boundary is a straight line through the midpoint of the two means (the priors are equal here).

**What to tune**

- **alpha:** (Categorical/Multinomial/Bernoulli) Larger → flatter likelihoods, softer posteriors. Grid-search on a log scale, e.g. $10^{-3}\ldots10$; text classifiers often prefer $\alpha<1$.
- **var_smoothing:** (Gaussian) The $\epsilon$ in $\sigma^2+\epsilon\max\operatorname{Var}$. Larger values widen every Gaussian, which smooths the boundary toward linear. Search $10^{-11}\ldots10^{-1}$.
- **priors / fit_prior:** Override $\pi_k$ when the deployment base rates differ from the training mix; "fit_prior=False" uses uniform priors.
- **event model:** Usually the biggest decision: counts → Multinomial/Complement, binary → Bernoulli, real → Gaussian (after a transform that makes features roughly normal), categories → Categorical. Mixed types: fit one model per block and add their log-likelihoods.
- **features:** Feature selection matters more than for most models: removing near-duplicate features reduces double counting.

## In code

```python
import numpy as np

class CategoricalNB:
    """Naive Bayes for categorical features, with Laplace/Lidstone smoothing alpha."""
    def __init__(self, alpha=1.0):
        self.alpha = alpha

    def fit(self, X, y):
        X, y = np.asarray(X), np.asarray(y)
        self.classes_ = np.unique(y)
        self.values_ = [np.unique(X[:, j]) for j in range(X.shape[1])]
        self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])
        # log_lik_[j][k, v] = log P(x_j = value v | class k)
        self.log_lik_ = []
        for j, vals in enumerate(self.values_):
            counts = np.array([[np.sum(X[y == c, j] == v) for v in vals] for c in self.classes_], float)
            probs = (counts + self.alpha) / (counts.sum(1, keepdims=True) + self.alpha * len(vals))
            with np.errstate(divide="ignore"):          # alpha=0 gives log(0) = -inf
                self.log_lik_.append(np.log(probs))
        return self

    def joint_log(self, x):
        """log P(y=k) + sum_j log P(x_j | y=k), for every class k."""
        s = self.log_prior_.copy()
        for j, v in enumerate(x):
            s += self.log_lik_[j][:, np.searchsorted(self.values_[j], v)]
        return s

    def predict_proba(self, X):
        S = np.array([self.joint_log(x) for x in np.asarray(X)])
        S -= S.max(1, keepdims=True)                     # log-sum-exp trick
        P = np.exp(S)
        return P / P.sum(1, keepdims=True)

    def predict(self, X):
        return self.classes_[self.predict_proba(X).argmax(1)]

class GaussianNB:
    def fit(self, X, y, eps=1e-9):
        self.classes_ = np.unique(y)
        self.mu_ = np.array([X[y == c].mean(0) for c in self.classes_])
        self.var_ = np.array([X[y == c].var(0) for c in self.classes_]) + eps * X.var(0).max()
        self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])
        return self

    def predict_proba(self, X):
        S = self.log_prior_ - 0.5 * (np.log(2 * np.pi * self.var_)[None]
                + (X[:, None, :] - self.mu_[None]) ** 2 / self.var_[None]).sum(-1)
        P = np.exp(S - S.max(1, keepdims=True))
        return P / P.sum(1, keepdims=True)

D = np.array("""
Sunny Hot High Weak No       Sunny Hot High Strong No      Overcast Hot High Weak Yes
Rain Mild High Weak Yes      Rain Cool Normal Weak Yes     Rain Cool Normal Strong No
Overcast Cool Normal Strong Yes  Sunny Mild High Weak No   Sunny Cool Normal Weak Yes
Rain Mild Normal Weak Yes    Sunny Mild Normal Strong Yes  Overcast Mild High Strong Yes
Overcast Hot Normal Weak Yes Rain Mild High Strong No""".split()).reshape(-1, 5)
X, y = D[:, :4], D[:, 4]

query = [["Sunny", "Cool", "High", "Strong"]]
for a in (0.0, 1.0):
    nb = CategoricalNB(alpha=a).fit(X, y)
    print(f"alpha={a}: classes {nb.classes_}, P = {nb.predict_proba(query).round(4)}, "
          f"predict {nb.predict(query)}, train acc {np.mean(nb.predict(X) == y):.3f}")

Xg = np.array([(2.2,6.1),(2.9,5.4),(3.4,6.8),(3.8,5.9),(2.6,7.0),(3.1,6.3),(4.0,6.6),(2.5,5.6),
               (5.0,3.2),(6.1,5.0),(7.8,3.6),(8.6,4.9),(6.8,2.4),(4.6,4.6),(7.2,5.6),(5.9,3.0)])
yg = np.repeat([0, 1], 8)
g = GaussianNB().fit(Xg, yg)
print("means", g.mu_.round(3).tolist(), "variances", g.var_.round(3).tolist())
print("P(class | x=(4.6, 5.4)) =", g.predict_proba(np.array([[4.6, 5.4]])).round(3))
```

From scratch with NumPy: categorical NB with smoothing, and Gaussian NB. Prints P = [[0.7954 0.2046]] for α = 0, [[0.7201 0.2799]] for α = 1, training accuracy 0.929, and P = [[0.247 0.753]] for the 2-D query.

```python
import pandas as pd
from sklearn.naive_bayes import CategoricalNB, GaussianNB, MultinomialNB
from sklearn.preprocessing import OrdinalEncoder
from sklearn.feature_extraction.text import CountVectorizer

df = pd.DataFrame({
    "Outlook":  "Sunny Sunny Overcast Rain Rain Rain Overcast Sunny Sunny Rain Sunny Overcast Overcast Rain".split(),
    "Temp":     "Hot Hot Hot Mild Cool Cool Cool Mild Cool Mild Mild Mild Hot Mild".split(),
    "Humidity": "High High High High Normal Normal Normal High Normal Normal Normal High Normal High".split(),
    "Wind":     "Weak Strong Weak Weak Weak Strong Strong Weak Weak Weak Strong Strong Weak Strong".split(),
    "Play":     "No No Yes Yes Yes No Yes No Yes Yes Yes Yes Yes No".split(),
})
enc = OrdinalEncoder()
X = enc.fit_transform(df.drop(columns="Play"))
nb = CategoricalNB(alpha=1.0).fit(X, df.Play)

new = pd.DataFrame([["Sunny", "Cool", "High", "Strong"]], columns=df.columns[:4])
print(nb.classes_, nb.predict_proba(enc.transform(new)).round(4))   # ['No' 'Yes'] [[0.7201 0.2799]]
print("train accuracy:", round(nb.score(X, df.Play), 3))                # 0.929 (day 6 is missed)

# Text: multinomial NB on word counts
docs = ["free money now", "free free prize", "win money free",
        "meeting at noon", "money for lunch", "noon meeting moved"]
spam = [1, 1, 1, 0, 0, 0]
cv = CountVectorizer()
mnb = MultinomialNB(alpha=1.0).fit(cv.fit_transform(docs), spam)
print(mnb.predict_proba(cv.transform(["free money meeting"])).round(4))  # [[0.2857 0.7143]]

# Continuous features: GaussianNB (var_smoothing adds a small fraction of the largest variance)
Xg = [(2.2,6.1),(2.9,5.4),(3.4,6.8),(3.8,5.9),(2.6,7.0),(3.1,6.3),(4.0,6.6),(2.5,5.6),
      (5.0,3.2),(6.1,5.0),(7.8,3.6),(8.6,4.9),(6.8,2.4),(4.6,4.6),(7.2,5.6),(5.9,3.0)]
g = GaussianNB().fit(Xg, [0] * 8 + [1] * 8)
print(g.predict_proba([[4.6, 5.4]]).round(3))                          # [[0.247 0.753]]
```

The same three examples with scikit-learn. Outputs are shown in the comments.

**Strengths**

- You have little labelled data, or very many features (text, bag-of-words, genomics): few parameters per feature means low variance.
- You need training and prediction to be extremely fast, incremental, or done on streaming data.
- You want a strong, explainable baseline: every prediction decomposes into per-feature log-odds.
- Classes may be added later — each class's model is fitted independently.

**Limitations**

- Features are strongly correlated or redundant: evidence is double-counted and probabilities become overconfident.
- You need calibrated probabilities (risk scores, expected cost) without a calibration step.
- Interactions matter (XOR-like patterns): the model cannot represent them at all.
- Data is plentiful: logistic regression, gradient boosting or a neural net will usually be more accurate.

- **Spam filtering:** The classic application (Sahami et al. 1998; Paul Graham's 'A Plan for Spam'): multinomial or Bernoulli NB over words, updated incrementally as users flag messages.
- **Text and document categorisation:** News topics, support-ticket routing, language identification — MultinomialNB/ComplementNB on TF or TF-IDF features is a hard-to-beat fast baseline.
- **Sentiment analysis:** Positive/negative reviews; NB features combined with an SVM (NBSVM) were a strong pre-transformer baseline.
- **Medical and fault diagnosis:** Symptoms → disease or alarm codes → fault, where each piece of evidence contributes an interpretable likelihood ratio.

> **Correlated features are counted twice:** Naive Bayes treats each feature as fresh evidence. Copy the Humidity column of the play-tennis data once more and its −0.875 vote is cast twice: for the Sunny/Cool/High/Strong day $P(\text{No}\mid x)$ goes $0.795\to0.903$, then $0.957$ and $0.982$ with three and four copies — with no new information at all. The ranking of classes is often still right, which is why NB classifies well, but its *probabilities* are pushed toward 0 and 1. Calibrate them ("CalibratedClassifierCV") before using them as risks.

> **Multiply in log space:** 200 likelihoods of 0.01 multiply to $10^{-400}$, which underflows a 64-bit float to exactly 0.0 — for every class, so the prediction becomes arbitrary. The sum of logs is a harmless $200\log 0.01=-921.03$. Every serious implementation works with log-probabilities and normalises with log-sum-exp.

> **Gaussian NB needs Gaussian-ish features:** A skewed feature (income, counts, durations) fitted with a single Gaussian per class gives poor likelihoods. Log- or power-transform it first, or discretise it into bins and use CategoricalNB. Scaling, on the other hand, does not matter: each feature's Gaussian adapts to its own units.

**Question:** With $\alpha=0$, a test email contains a word that never appeared in any training spam. What is its posterior probability of spam?

- A. Exactly 0, whatever the other words say
- B. About 0.5
- C. Unchanged — unseen words are ignored
- D. Slightly lower than without the word

**Answer:** A. Exactly 0, whatever the other words say — The spam likelihood of that word is $0/N_{\text{spam}}=0$, and one zero factor makes the whole product zero. (Words absent from the *entire* vocabulary are dropped by the vectoriser; the problem is a known word with a zero count in one class.) Smoothing with $\alpha>0$ fixes it.

**Question:** In the spam corpus above (spam has 9 tokens, the vocabulary has 11 words, "free" appears 4 times in spam), what is $P(\text{free}\mid\text{spam})$ with $\alpha=1$?

- A. 4/9 ≈ 0.444
- B. 5/20 = 0.25
- C. 5/10 = 0.5
- D. 4/11 ≈ 0.364

**Answer:** B. 5/20 = 0.25 — $(N_{\text{spam,free}}+\alpha)/(N_{\text{spam}}+\alpha|V|)=(4+1)/(9+11)=5/20=0.25$. The denominator adds $\alpha$ for every word in the vocabulary so the 11 probabilities still sum to 1.

**Question:** You duplicate one feature column three times before training naive Bayes. What changes?

- A. Nothing — the model detects the duplicates
- B. That feature's log-likelihood ratio is added three times, so posteriors become more extreme
- C. Training fails because the features are collinear
- D. The prior is multiplied by three

**Answer:** B. That feature's log-likelihood ratio is added three times, so posteriors become more extreme — Under the independence assumption each copy is new evidence, so its vote is tripled. On the play-tennis day, three copies of Humidity raise $P(\text{No}\mid x)$ from 0.795 to 0.957. Unlike linear regression, nothing breaks numerically — the model just becomes overconfident.

**Question:** Gaussian naive Bayes with the same variance for each feature in every class has a decision boundary that is…

- A. Linear
- B. Quadratic (a conic)
- C. Axis-aligned boxes
- D. Arbitrary

**Answer:** A. Linear — In $\log\frac{P(x\mid A)}{P(x\mid B)}$ the $x_j^2/(2\sigma_j^2)$ terms are identical for both classes and cancel, leaving a function linear in $x$. With per-class variances they do not cancel and the boundary is quadratic — as the Gaussian tab of the figure shows.

**Key points**

- Naive Bayes applies Bayes' rule with the assumption that features are independent given the class: $P(y\mid x)\propto P(y)\prod_j P(x_j\mid y)$.
- Training is closed-form maximum likelihood: count tables (categorical, multinomial, Bernoulli) or per-class means and variances (Gaussian) — one pass, O(nd).
- Work in log space: each feature adds a log-likelihood ratio to the prior log-odds, which makes predictions explainable and avoids underflow.
- Smooth counts with $\alpha$ (Laplace = 1): otherwise one unseen value forces a posterior of exactly 0.
- Choose the event model to match the features; Gaussian NB is QDA with diagonal covariances, and discrete NB is a linear classifier like logistic regression.
- It ranks classes well even when independence is false, but its probabilities are overconfident — calibrate them before trusting them.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
