Maximum Likelihood Estimation

Pick the parameters that make the observed data most probable; least squares is MLE under Gaussian noise.

6 min read355 wordscore3 viewsupdated 2026-10-03

Definition

Maximum likelihood estimation (MLE) chooses the parameter values θ\theta under which the observed data are most probable: θ^=arg⁡max⁡θ p(x1,…,xn∣θ)\hat\theta=\arg\max_\theta\,p(x_1,\dots,x_n\mid\theta). Read as a function of θ\theta with the data held fixed, p(data∣θ)p(\text{data}\mid\theta) is the likelihood L(θ)L(\theta).

At a glance

Notation
L(θ)=∏ip(xi∣θ)L(\theta)=\prod_i p(x_i\mid\theta), log-likelihood ℓ(θ)=log⁡L(θ)\ell(\theta)=\log L(\theta)
Input → output
data and a parametric model p(x∣θ)p(x\mid\theta) → estimate θ^\hat\theta
Key property
consistent and asymptotically efficient: as n→∞n\to\infty, θ^→θ\hat\theta\to\theta with the smallest achievable variance (under regularity conditions)
Prerequisites
probability distributions, logarithms, derivatives
Used in
least squares, logistic regression, cross-entropy training, Gaussian mixtures

The intuition

A model with parameters θ\theta assigns a probability (or density) to every possible dataset. Some values of θ\theta make the data you actually observed plausible; others make it nearly impossible. MLE turns which parameters? into an optimization: move θ\theta until the observed data are as likely as they can be.

The likelihood is not a probability distribution over θ\theta — it need not integrate to 1. It only ranks parameter values by how well they explain the data.

The math

Assume the observations are independent and identically distributed (i.i.d.). The joint density then factorizes, and the log turns the product into a sum — numerically stable and easy to differentiate. Because log⁡\log is strictly increasing, maximizing ℓ\ell maximizes LL:

θ^=arg⁡max⁡θ ℓ(θ)=arg⁡max⁡θ ∑i=1nlog⁡p(xi∣θ)\hat\theta=\arg\max_\theta\ \ell(\theta)=\arg\max_\theta\ \sum_{i=1}^{n}\log p(x_i\mid\theta)
p(x∣θ)p(x\mid\theta)
model density of one observation
ℓ(θ)\ell(\theta)
log-likelihood

Gaussian. For xi∼N(μ,σ2)x_i\sim\mathcal N(\mu,\sigma^2) the log-likelihood is

ℓ(μ,σ)=−n2log⁡(2πσ2)−12σ2∑i=1n(xi−μ)2\ell(\mu,\sigma)=-\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i-\mu)^2

Setting ∂ℓ/∂μ=0\partial\ell/\partial\mu=0 and ∂ℓ/∂σ=0\partial\ell/\partial\sigma=0 gives

μ^=xˉ=1n∑ixi,σ^2=1n∑i(xi−xˉ)2\hat\mu=\bar x=\frac1n\sum_i x_i,\qquad \hat\sigma^2=\frac1n\sum_i(x_i-\bar x)^2

Note the 1n\frac1n: the MLE of the variance is biased low by a factor n−1n\frac{n-1}{n}. The unbiased sample variance s2s^2 divides by n−1n-1.

Least squares is MLE. If yi=fw(xi)+εiy_i=f_w(x_i)+\varepsilon_i with εi∼N(0,σ2)\varepsilon_i\sim\mathcal N(0,\sigma^2), then ℓ(w)=const−12σ2∑i(yi−fw(xi))2\ell(w)=\text{const}-\frac{1}{2\sigma^2}\sum_i\big(y_i-f_w(x_i)\big)^2. Maximizing it over ww is exactly minimizing the sum of squared errors. Bernoulli labels give log-loss the same way, and categorical labels give cross-entropy.

See it

012345678910bar height = N(xᵢ | μ, σ²) · likelihood = product of barslog-likelihood vs μ (σ = 2.40)μ̂ = x̄ = 6.18
log L = -27.17 (max -16.99 at μ̂ = 6.18, σ̂ = 1.32)
fig.Drag μ and σ: the likelihood is the product of the orange bars. ‘Fit by MLE’ moves to μ̂ = x̄ = 6.18, σ̂ = 1.32, where log L peaks at −16.99. The coin tab shows L(p) = pʰ(1 − p)ᵗ peaking at h/n.

Worked example

Coin. 7 heads in 10 flips: ℓ(p)=7log⁡p+3log⁡(1−p)\ell(p)=7\log p+3\log(1-p), and ℓ′(p)=7p−31−p=0⇒p^=0.7\ell'(p)=\frac7p-\frac{3}{1-p}=0\Rightarrow\hat p=0.7. Then ℓ(0.7)=−6.109\ell(0.7)=-6.109 versus ℓ(0.5)=−6.931\ell(0.5)=-6.931: the data are e0.823≈2.28e^{0.823}\approx2.28 times more likely under p=0.7p=0.7 than under a fair coin.

Gaussian. Data 2,4,62, 4, 6: μ^=4\hat\mu=4 and σ^2=(−2)2+02+223=83≈2.67\hat\sigma^2=\frac{(-2)^2+0^2+2^2}{3}=\frac83\approx2.67, while the unbiased estimate is s2=82=4s^2=\frac82=4.

In code

python
1import numpy as np
2from scipy import stats, optimize
3
4x = np.array([2.0, 4.0, 6.0])
5mu_hat, sigma_hat = x.mean(), x.std(ddof=0) # closed-form MLE: 4.0, 1.633
6
7# The same answer by numerically maximizing the log-likelihood
8nll = lambda t: -stats.norm.logpdf(x, t[0], np.exp(t[1])).sum()
9res = optimize.minimize(nll, x0=[0.0, 0.0])
10res.x[0], np.exp(res.x[1]) # ≈ 4.0, 1.633
11
12stats.norm.fit(x) # SciPy's MLE: (4.0, 1.633)
Closed form, numerical optimization and SciPy's fit agree. Optimizing log σ keeps σ positive.

Where it's used

Training ML models

Minimizing MSE, log-loss or cross-entropy is maximizing a Gaussian, Bernoulli or categorical likelihood.

Fitting distributions

Model waiting times (exponential), counts (Poisson) or durations (log-normal), then simulate or price risk.

Model comparison

AIC =2k−2ℓ(θ^)=2k-2\ell(\hat\theta) and BIC =klog⁡n−2ℓ(θ^)=k\log n-2\ell(\hat\theta) trade fit against the number of parameters kk.

Latent-variable models

EM for Gaussian mixtures and hidden Markov models maximizes a likelihood with unobserved labels.

Check yourself

You observe counts 2,3,72, 3, 7 from a Poisson(λ\lambda) distribution. The MLE λ^\hat\lambda is…

Check yourself

Minimizing mean squared error in regression is maximum likelihood when the noise is…

Remember

  • ✓θ^=arg⁡max⁡θ∑ilog⁡p(xi∣θ)\hat\theta=\arg\max_\theta\sum_i\log p(x_i\mid\theta) for i.i.d. data.
  • ✓Gaussian: μ^=xˉ\hat\mu=\bar x, σ^2=1n∑(xi−xˉ)2\hat\sigma^2=\frac1n\sum(x_i-\bar x)^2 — biased by n−1n\frac{n-1}{n}.
  • ✓Least squares = Gaussian MLE; log-loss = Bernoulli MLE.
  • ✓MLE overfits small samples — add a prior (MAP) or regularize.
← back to Mathematics roadmap