Definition
Naive Bayes is a family of probabilistic classifiers that model how each class generates data — a prior and, for every feature separately, a distribution — and then use Bayes' rule to turn that model around into . The "naive" part is the assumption that the features are conditionally independent given the class, which lets the joint likelihood factor into a product of one-feature pieces that can each be estimated by counting.
At a glance
- Task
- Classification (binary or multi-class), with class probabilities
- Input → output
- feature vector → posterior for every class; predict the largest
- Learns
- class priors and one small distribution per (feature, class): a count table, a Gaussian , or word frequencies
- Training
- one pass over the data: — no optimisation, no iterations
- Prediction
- : add log-likelihoods for each of classes
- Hyperparameters
- smoothing (counts), "var_smoothing" (Gaussian), class priors, and the choice of per-feature distribution
- Assumes
- features independent given the class; each feature follows the chosen family (categorical, Gaussian, multinomial, Bernoulli)
The problem
Bayes' rule tells us exactly how to classify: pick the class with the largest . The trouble is — the probability of an entire feature vector. With just 30 yes/no features there are possible vectors per class; we would need billions of examples to estimate their probabilities, and almost every new input would be a vector we have never seen, with estimated probability 0.
Discriminative models such as logistic regression dodge the problem by modelling directly, but they need iterative optimisation and enough data to pin down a weight per feature. Naive Bayes takes the opposite route: keep the generative story, but make one bold simplification so that needs only one-dimensional estimates instead of one -dimensional one.
The idea
Pretend that, once you know the class, the features tell you nothing about each other. Knowing an email is spam, whether it contains "free" says nothing extra about whether it contains "money". Then the probability of the whole vector is just the product of the individual feature probabilities, and each of those is estimated from the training rows of that class — usually by counting.
To classify, every class starts with its prior and then each feature votes: its likelihood under that class multiplies in. In log space the votes add up, so naive Bayes is an evidence ledger — each feature contributes a fixed amount of log-odds for or against each class, and the class with the largest total wins.
Vocabulary
- Prior
- : how common class is before looking at the features — the fraction of training rows with label
- Likelihood
- : how probable feature value is among examples of class
- Posterior
- : the class probability after seeing the features; what we predict from
- Evidence
- : the normaliser that makes posteriors sum to 1; it is the same for every class, so the arg-max ignores it
- Conditional independence
- : independence within each class. Features can still be strongly correlated overall (through the class)
- Generative model
- one that models — how data is produced — rather than only the boundary
- Smoothing (α)
- pseudo-counts added to every count so that an unseen value gets a small probability instead of zero
- Event model
- the per-feature distribution family: categorical, Gaussian, multinomial (word counts) or Bernoulli (word present/absent)
The math
Start from Bayes' rule for a class and a feature vector :
- prior of class
- class-conditional likelihood of the whole vector
- evidence , identical for all classes
The chain rule gives exactly. The naive assumption drops every conditioning variable except the class, , so
- prior , estimated as
- number of features
- predicted class (the maximum a posteriori class)
We take logs for two reasons: the product of hundreds of small probabilities underflows to 0 in floating point, and sums are cheaper and easier to read than products. Posteriors are recovered with the log-sum-exp trick: subtract the largest class score, exponentiate, normalise.
Training is counting: the maximum-likelihood estimate
For a categorical feature with values , the model for class is a probability vector . Let be the number of class- rows whose feature equals . The log-likelihood of the training data that involves this vector is , to be maximised subject to . With a Lagrange multiplier :
- count of class- rows with
- number of class- rows; the constraint forces
- estimated
So the maximum-likelihood fit is literally a frequency table, and because the log-likelihood splits into one independent term per (class, feature), every table is fitted on its own in a single pass. The same argument gives .
Smoothing. If a value never occurs with class , and any input with that value gets posterior 0 for class , no matter how strongly every other feature points to it. The fix adds pseudo-counts to every cell:
- smoothing strength: is Laplace smoothing, Lidstone; is the raw MLE
- number of distinct values of feature (so the probabilities still sum to 1)
This is the posterior mean of under a symmetric Dirichlet() prior: imaginary observations of each value. Mathematically shrinks every estimate toward the uniform , and the shrinkage weight is — large when the class is rare, negligible when . Larger therefore means flatter likelihood ratios, less extreme posteriors, and more bias; smaller means more variance.
Continuous features: Gaussian naive Bayes
For a real-valued feature, the usual choice is one Gaussian per (class, feature). The maximum-likelihood estimates are again closed-form — the class's sample mean and (biased, divide-by-) variance of that feature:
- mean and variance of feature within class
- feature of training row
Multiplying one-dimensional Gaussians gives a -dimensional Gaussian with a diagonal covariance: its contours are ellipses aligned with the axes. So Gaussian NB is exactly quadratic discriminant analysis with every off-diagonal covariance forced to zero; tie the variances across classes and it becomes a diagonal version of linear discriminant analysis (see those two articles for the full-covariance story). scikit-learn's "var_smoothing" adds to every (default ) so a constant feature in some class does not produce a zero variance and an infinite density.
Naive Bayes is a linear classifier (for discrete features). For two classes, the log-odds is a sum of per-feature terms: . With one-hot or count features each term is linear in the features, so the decision boundary is a hyperplane — the same functional form as logistic regression.
Worked example: play tennis, by hand
The same 14 days used in the decision-trees article: four categorical features and whether tennis was played (9 Yes, 5 No). We classify a new day: Sunny, Cool, High humidity, Strong wind.
| Day | Outlook | Temp | Humidity | Wind | Play |
|---|---|---|---|---|---|
| 1 | Sunny | Hot | High | Weak | No |
| 2 | Sunny | Hot | High | Strong | No |
| 3 | Overcast | Hot | High | Weak | Yes |
| 4 | Rain | Mild | High | Weak | Yes |
| 5 | Rain | Cool | Normal | Weak | Yes |
| 6 | Rain | Cool | Normal | Strong | No |
| 7 | Overcast | Cool | Normal | Strong | Yes |
| 8 | Sunny | Mild | High | Weak | No |
| 9 | Sunny | Cool | Normal | Weak | Yes |
| 10 | Rain | Mild | Normal | Weak | Yes |
| 11 | Sunny | Mild | Normal | Strong | Yes |
| 12 | Overcast | Mild | High | Strong | Yes |
| 13 | Overcast | Hot | Normal | Weak | Yes |
| 14 | Rain | Mild | High | Strong | No |
Training is one pass that fills the count table below. That is the entire model.
| Feature = value | Yes count | No count | P(value | Yes) | P(value | No) |
|---|---|---|---|---|
| Outlook = Sunny | 2 | 3 | 2/9 = 0.222 | 3/5 = 0.600 |
| Outlook = Overcast | 4 | 0 | 4/9 = 0.444 | 0/5 = 0 |
| Outlook = Rain | 3 | 2 | 3/9 = 0.333 | 2/5 = 0.400 |
| Temp = Hot | 2 | 2 | 2/9 = 0.222 | 2/5 = 0.400 |
| Temp = Mild | 4 | 2 | 4/9 = 0.444 | 2/5 = 0.400 |
| Temp = Cool | 3 | 1 | 3/9 = 0.333 | 1/5 = 0.200 |
| Humidity = High | 3 | 4 | 3/9 = 0.333 | 4/5 = 0.800 |
| Humidity = Normal | 6 | 1 | 6/9 = 0.667 | 1/5 = 0.200 |
| Wind = Weak | 6 | 2 | 6/9 = 0.667 | 2/5 = 0.400 |
| Wind = Strong | 3 | 3 | 3/9 = 0.333 | 3/5 = 0.600 |
Classify (Sunny, Cool, High, Strong) with raw counts (α = 0)
- 1Priors: , .
- 2Yes score: .
- 3No score: .
- 4Normalise: , .
- 5Predict No. The prior favoured Yes, but Sunny, High humidity and Strong wind are all much more typical of No days; only Cool argues for Yes.
The same computation in log-odds makes the "vote" of each feature explicit. Each row is ; positive favours Yes.
| Term | Yes | No | log ratio (Yes : No) |
|---|---|---|---|
| prior | 9/14 | 5/14 | +0.588 |
| Outlook = Sunny | 2/9 | 3/5 | −0.993 |
| Temp = Cool | 3/9 | 1/5 | +0.511 |
| Humidity = High | 3/9 | 4/5 | −0.875 |
| Wind = Strong | 3/9 | 3/5 | −0.588 |
| total log-odds | −1.358 |
The same day with Laplace smoothing (α = 1)
- 1Each Yes likelihood becomes and each No likelihood , with values for Outlook, Temp, Humidity, Wind.
- 2Yes: . No: .
- 3. Still No, but less confident: smoothing pulled every ratio toward 1, and the small No class (5 rows) is pulled hardest.
On its own 14 training days the model (with either α) gets 13 right; it misses day 6 (Rain, Cool, Normal, Strong → No), where the Cool and Normal votes for Yes outweigh Rain and Strong.
The algorithm, step by step
fit(X, y) and predict(x)
- 1Priors. For each class : (or use priors you supply, e.g. known base rates).
- 2Likelihood tables. For each class and feature, compute sufficient statistics in one pass: value counts (categorical), token counts (multinomial), presence counts (Bernoulli), or mean and variance (Gaussian).
- 3Smooth and log. Turn counts into (or add "var_smoothing" to variances) and store the log-probabilities.
- 4Score. For a new : — a table lookup and an addition per feature.
- 5Normalise. with ; predict .
Cost
- Training time
- — one pass; trivially parallel and incremental (add new rows' counts with "partial_fit")
- Prediction time
- per example; for sparse text only the non-zero words cost anything
- Memory
- for count tables, for Gaussians — independent of
Variants and hyperparameters
The variants differ only in the event model ; the prior and the arg-max are the same.
| Variant | Feature type | P(x_j | y=k) | Typical use |
|---|---|---|---|
| CategoricalNB | categories | tabular categorical data | |
| GaussianNB | real numbers | sensor readings, quick baselines | |
| MultinomialNB | counts (word frequencies) | , | text classification, spam |
| BernoulliNB | binary (word present?) | short texts; absence counts as evidence | |
| ComplementNB | counts | multinomial estimated from all other classes | imbalanced text classes |
Multinomial NB on a tiny spam corpus. Three spam messages ("free money now", "free free prize", "win money free") and three ham ("meeting at noon", "money for lunch", "noon meeting moved"). Each class has tokens and the vocabulary has words. With , :
| Word | count in spam | count in ham | P(w | spam) | P(w | ham) |
|---|---|---|---|---|
| free | 4 | 0 | 5/20 = 0.25 | 1/20 = 0.05 |
| money | 2 | 1 | 3/20 = 0.15 | 2/20 = 0.10 |
| meeting | 0 | 2 | 1/20 = 0.05 | 3/20 = 0.15 |
Note that the multinomial model ignores the 8 vocabulary words absent from the message; Bernoulli NB multiplies in for each of them, and on this corpus gives P(spam) = 0.842 instead.
| Variances | Class A: μ, σ² | Class B: μ, σ² | log score A | log score B | Posterior |
|---|---|---|---|---|---|
| per class | (3.06, 6.21), (0.35, 0.28) | (6.50, 4.04), (1.63, 1.13) | −0.693 − 3.732 − 1.453 = −5.878 | −0.693 − 2.270 − 1.800 = −4.763 | P(B) = 0.753 |
| shared | (3.06, 6.21), (0.99, 0.71) | (6.50, 4.04), (0.99, 0.71) | −4.011 | −5.481 | P(A) = 0.813 |
With per-class variances, the wide class B has a density that decays slowly, so it claims the space between the classes and the boundary curves around the tight class A. With a shared variance the quadratic terms cancel in the log-ratio and the boundary is a straight line through the midpoint of the two means (the priors are equal here).
What to tune
- alpha
- (Categorical/Multinomial/Bernoulli) Larger → flatter likelihoods, softer posteriors. Grid-search on a log scale, e.g. ; text classifiers often prefer .
- var_smoothing
- (Gaussian) The in . Larger values widen every Gaussian, which smooths the boundary toward linear. Search .
- priors / fit_prior
- Override when the deployment base rates differ from the training mix; "fit_prior=False" uses uniform priors.
- event model
- Usually the biggest decision: counts → Multinomial/Complement, binary → Bernoulli, real → Gaussian (after a transform that makes features roughly normal), categories → Categorical. Mixed types: fit one model per block and add their log-likelihoods.
- features
- Feature selection matters more than for most models: removing near-duplicate features reduces double counting.
In code
1import numpy as np23class CategoricalNB:4 """Naive Bayes for categorical features, with Laplace/Lidstone smoothing alpha."""5 def __init__(self, alpha=1.0):6 self.alpha = alpha78 def fit(self, X, y):9 X, y = np.asarray(X), np.asarray(y)10 self.classes_ = np.unique(y)11 self.values_ = [np.unique(X[:, j]) for j in range(X.shape[1])]12 self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])13 # log_lik_[j][k, v] = log P(x_j = value v | class k)14 self.log_lik_ = []15 for j, vals in enumerate(self.values_):16 counts = np.array([[np.sum(X[y == c, j] == v) for v in vals] for c in self.classes_], float)17 probs = (counts + self.alpha) / (counts.sum(1, keepdims=True) + self.alpha * len(vals))18 with np.errstate(divide="ignore"): # alpha=0 gives log(0) = -inf19 self.log_lik_.append(np.log(probs))20 return self2122 def joint_log(self, x):23 """log P(y=k) + sum_j log P(x_j | y=k), for every class k."""24 s = self.log_prior_.copy()25 for j, v in enumerate(x):26 s += self.log_lik_[j][:, np.searchsorted(self.values_[j], v)]27 return s2829 def predict_proba(self, X):30 S = np.array([self.joint_log(x) for x in np.asarray(X)])31 S -= S.max(1, keepdims=True) # log-sum-exp trick32 P = np.exp(S)33 return P / P.sum(1, keepdims=True)3435 def predict(self, X):36 return self.classes_[self.predict_proba(X).argmax(1)]3738class GaussianNB:39 def fit(self, X, y, eps=1e-9):40 self.classes_ = np.unique(y)41 self.mu_ = np.array([X[y == c].mean(0) for c in self.classes_])42 self.var_ = np.array([X[y == c].var(0) for c in self.classes_]) + eps * X.var(0).max()43 self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])44 return self4546 def predict_proba(self, X):47 S = self.log_prior_ - 0.5 * (np.log(2 * np.pi * self.var_)[None]48 + (X[:, None, :] - self.mu_[None]) ** 2 / self.var_[None]).sum(-1)49 P = np.exp(S - S.max(1, keepdims=True))50 return P / P.sum(1, keepdims=True)5152D = np.array("""53Sunny Hot High Weak No Sunny Hot High Strong No Overcast Hot High Weak Yes54Rain Mild High Weak Yes Rain Cool Normal Weak Yes Rain Cool Normal Strong No55Overcast Cool Normal Strong Yes Sunny Mild High Weak No Sunny Cool Normal Weak Yes56Rain Mild Normal Weak Yes Sunny Mild Normal Strong Yes Overcast Mild High Strong Yes57Overcast Hot Normal Weak Yes Rain Mild High Strong No""".split()).reshape(-1, 5)58X, y = D[:, :4], D[:, 4]5960query = [["Sunny", "Cool", "High", "Strong"]]61for a in (0.0, 1.0):62 nb = CategoricalNB(alpha=a).fit(X, y)63 print(f"alpha={a}: classes {nb.classes_}, P = {nb.predict_proba(query).round(4)}, "64 f"predict {nb.predict(query)}, train acc {np.mean(nb.predict(X) == y):.3f}")6566Xg = np.array([(2.2,6.1),(2.9,5.4),(3.4,6.8),(3.8,5.9),(2.6,7.0),(3.1,6.3),(4.0,6.6),(2.5,5.6),67 (5.0,3.2),(6.1,5.0),(7.8,3.6),(8.6,4.9),(6.8,2.4),(4.6,4.6),(7.2,5.6),(5.9,3.0)])68yg = np.repeat([0, 1], 8)69g = GaussianNB().fit(Xg, yg)70print("means", g.mu_.round(3).tolist(), "variances", g.var_.round(3).tolist())71print("P(class | x=(4.6, 5.4)) =", g.predict_proba(np.array([[4.6, 5.4]])).round(3))1import pandas as pd2from sklearn.naive_bayes import CategoricalNB, GaussianNB, MultinomialNB3from sklearn.preprocessing import OrdinalEncoder4from sklearn.feature_extraction.text import CountVectorizer56df = pd.DataFrame({7 "Outlook": "Sunny Sunny Overcast Rain Rain Rain Overcast Sunny Sunny Rain Sunny Overcast Overcast Rain".split(),8 "Temp": "Hot Hot Hot Mild Cool Cool Cool Mild Cool Mild Mild Mild Hot Mild".split(),9 "Humidity": "High High High High Normal Normal Normal High Normal Normal Normal High Normal High".split(),10 "Wind": "Weak Strong Weak Weak Weak Strong Strong Weak Weak Weak Strong Strong Weak Strong".split(),11 "Play": "No No Yes Yes Yes No Yes No Yes Yes Yes Yes Yes No".split(),12})13enc = OrdinalEncoder()14X = enc.fit_transform(df.drop(columns="Play"))15nb = CategoricalNB(alpha=1.0).fit(X, df.Play)1617new = pd.DataFrame([["Sunny", "Cool", "High", "Strong"]], columns=df.columns[:4])18print(nb.classes_, nb.predict_proba(enc.transform(new)).round(4)) # ['No' 'Yes'] [[0.7201 0.2799]]19print("train accuracy:", round(nb.score(X, df.Play), 3)) # 0.929 (day 6 is missed)2021# Text: multinomial NB on word counts22docs = ["free money now", "free free prize", "win money free",23 "meeting at noon", "money for lunch", "noon meeting moved"]24spam = [1, 1, 1, 0, 0, 0]25cv = CountVectorizer()26mnb = MultinomialNB(alpha=1.0).fit(cv.fit_transform(docs), spam)27print(mnb.predict_proba(cv.transform(["free money meeting"])).round(4)) # [[0.2857 0.7143]]2829# Continuous features: GaussianNB (var_smoothing adds a small fraction of the largest variance)30Xg = [(2.2,6.1),(2.9,5.4),(3.4,6.8),(3.8,5.9),(2.6,7.0),(3.1,6.3),(4.0,6.6),(2.5,5.6),31 (5.0,3.2),(6.1,5.0),(7.8,3.6),(8.6,4.9),(6.8,2.4),(4.6,4.6),(7.2,5.6),(5.9,3.0)]32g = GaussianNB().fit(Xg, [0] * 8 + [1] * 8)33print(g.predict_proba([[4.6, 5.4]]).round(3)) # [[0.247 0.753]]Use when
- +You have little labelled data, or very many features (text, bag-of-words, genomics): few parameters per feature means low variance.
- +You need training and prediction to be extremely fast, incremental, or done on streaming data.
- +You want a strong, explainable baseline: every prediction decomposes into per-feature log-odds.
- +Classes may be added later — each class's model is fitted independently.
Avoid when
- −Features are strongly correlated or redundant: evidence is double-counted and probabilities become overconfident.
- −You need calibrated probabilities (risk scores, expected cost) without a calibration step.
- −Interactions matter (XOR-like patterns): the model cannot represent them at all.
- −Data is plentiful: logistic regression, gradient boosting or a neural net will usually be more accurate.
Where it's used
Spam filtering
The classic application (Sahami et al. 1998; Paul Graham's 'A Plan for Spam'): multinomial or Bernoulli NB over words, updated incrementally as users flag messages.
Text and document categorisation
News topics, support-ticket routing, language identification — MultinomialNB/ComplementNB on TF or TF-IDF features is a hard-to-beat fast baseline.
Sentiment analysis
Positive/negative reviews; NB features combined with an SVM (NBSVM) were a strong pre-transformer baseline.
Medical and fault diagnosis
Symptoms → disease or alarm codes → fault, where each piece of evidence contributes an interpretable likelihood ratio.
Check yourself
With , a test email contains a word that never appeared in any training spam. What is its posterior probability of spam?
Check yourself
In the spam corpus above (spam has 9 tokens, the vocabulary has 11 words, "free" appears 4 times in spam), what is with ?
Check yourself
You duplicate one feature column three times before training naive Bayes. What changes?
Check yourself
Gaussian naive Bayes with the same variance for each feature in every class has a decision boundary that is…
Remember
- ✓Naive Bayes applies Bayes' rule with the assumption that features are independent given the class: .
- ✓Training is closed-form maximum likelihood: count tables (categorical, multinomial, Bernoulli) or per-class means and variances (Gaussian) — one pass, O(nd).
- ✓Work in log space: each feature adds a log-likelihood ratio to the prior log-odds, which makes predictions explainable and avoids underflow.
- ✓Smooth counts with (Laplace = 1): otherwise one unseen value forces a posterior of exactly 0.
- ✓Choose the event model to match the features; Gaussian NB is QDA with diagonal covariances, and discrete NB is a linear classifier like logistic regression.
- ✓It ranks classes well even when independence is false, but its probabilities are overconfident — calibrate them before trusting them.