Definition
Maximum likelihood estimation (MLE) chooses the parameter values under which the observed data are most probable: . Read as a function of with the data held fixed, is the likelihood .
At a glance
- Notation
- , log-likelihood
- Input → output
- data and a parametric model → estimate
- Key property
- consistent and asymptotically efficient: as , with the smallest achievable variance (under regularity conditions)
- Prerequisites
- probability distributions, logarithms, derivatives
- Used in
- least squares, logistic regression, cross-entropy training, Gaussian mixtures
The intuition
A model with parameters assigns a probability (or density) to every possible dataset. Some values of make the data you actually observed plausible; others make it nearly impossible. MLE turns which parameters? into an optimization: move until the observed data are as likely as they can be.
The likelihood is not a probability distribution over — it need not integrate to 1. It only ranks parameter values by how well they explain the data.
The math
Assume the observations are independent and identically distributed (i.i.d.). The joint density then factorizes, and the log turns the product into a sum — numerically stable and easy to differentiate. Because is strictly increasing, maximizing maximizes :
- model density of one observation
- log-likelihood
Gaussian. For the log-likelihood is
Setting and gives
Note the : the MLE of the variance is biased low by a factor . The unbiased sample variance divides by .
Least squares is MLE. If with , then . Maximizing it over is exactly minimizing the sum of squared errors. Bernoulli labels give log-loss the same way, and categorical labels give cross-entropy.
See it
Worked example
Coin. 7 heads in 10 flips: , and . Then versus : the data are times more likely under than under a fair coin.
Gaussian. Data : and , while the unbiased estimate is .
In code
1import numpy as np2from scipy import stats, optimize34x = np.array([2.0, 4.0, 6.0])5mu_hat, sigma_hat = x.mean(), x.std(ddof=0) # closed-form MLE: 4.0, 1.63367# The same answer by numerically maximizing the log-likelihood8nll = lambda t: -stats.norm.logpdf(x, t[0], np.exp(t[1])).sum()9res = optimize.minimize(nll, x0=[0.0, 0.0])10res.x[0], np.exp(res.x[1]) # ≈ 4.0, 1.6331112stats.norm.fit(x) # SciPy's MLE: (4.0, 1.633)Where it's used
Training ML models
Minimizing MSE, log-loss or cross-entropy is maximizing a Gaussian, Bernoulli or categorical likelihood.
Fitting distributions
Model waiting times (exponential), counts (Poisson) or durations (log-normal), then simulate or price risk.
Model comparison
AIC and BIC trade fit against the number of parameters .
Latent-variable models
EM for Gaussian mixtures and hidden Markov models maximizes a likelihood with unobserved labels.
Check yourself
You observe counts from a Poisson() distribution. The MLE is…
Check yourself
Minimizing mean squared error in regression is maximum likelihood when the noise is…
Remember
- ✓ for i.i.d. data.
- ✓Gaussian: , — biased by .
- ✓Least squares = Gaussian MLE; log-loss = Bernoulli MLE.
- ✓MLE overfits small samples — add a prior (MAP) or regularize.