# Maximum Likelihood Estimation

> Pick the parameters that make the observed data most probable; least squares is MLE under Gaussian noise.

Source: https://www.mlfoundry.in/learn/maximum-likelihood  
Section: Mathematics › Statistics  
Level: core · 6 min read · updated 2026-10-03

**Maximum likelihood estimation (MLE):** chooses the parameter values $\theta$ under which the observed data are most probable: $\hat\theta=\arg\max_\theta\,p(x_1,\dots,x_n\mid\theta)$. Read as a function of $\theta$ with the data held fixed, $p(\text{data}\mid\theta)$ is the **likelihood** $L(\theta)$.

- **Notation:** $L(\theta)=\prod_i p(x_i\mid\theta)$, log-likelihood $\ell(\theta)=\log L(\theta)$
- **Input → output:** data and a parametric model $p(x\mid\theta)$ → estimate $\hat\theta$
- **Key property:** consistent and asymptotically efficient: as $n\to\infty$, $\hat\theta\to\theta$ with the smallest achievable variance (under regularity conditions)
- **Prerequisites:** probability distributions, logarithms, derivatives
- **Used in:** least squares, logistic regression, cross-entropy training, Gaussian mixtures

## The intuition

A model with parameters $\theta$ assigns a probability (or density) to every possible dataset. Some values of $\theta$ make the data you actually observed plausible; others make it nearly impossible. MLE turns *which parameters?* into an optimization: move $\theta$ until the observed data are as likely as they can be.

The likelihood is **not** a probability distribution over $\theta$ — it need not integrate to 1. It only ranks parameter values by how well they explain the data.

## The math

Assume the observations are independent and identically distributed (i.i.d.). The joint density then factorizes, and the log turns the product into a sum — numerically stable and easy to differentiate. Because $\log$ is strictly increasing, maximizing $\ell$ maximizes $L$:

$$
\hat\theta=\arg\max_\theta\ \ell(\theta)=\arg\max_\theta\ \sum_{i=1}^{n}\log p(x_i\mid\theta)
$$

- $p(x\mid\theta)$: model density of one observation
- $\ell(\theta)$: log-likelihood

**Gaussian.** For $x_i\sim\mathcal N(\mu,\sigma^2)$ the log-likelihood is

$$
\ell(\mu,\sigma)=-\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i-\mu)^2
$$

Setting $\partial\ell/\partial\mu=0$ and $\partial\ell/\partial\sigma=0$ gives

$$
\hat\mu=\bar x=\frac1n\sum_i x_i,\qquad \hat\sigma^2=\frac1n\sum_i(x_i-\bar x)^2
$$

Note the $\frac1n$: the MLE of the variance is biased low by a factor $\frac{n-1}{n}$. The unbiased sample variance $s^2$ divides by $n-1$.

**Least squares is MLE.** If $y_i=f_w(x_i)+\varepsilon_i$ with $\varepsilon_i\sim\mathcal N(0,\sigma^2)$, then $\ell(w)=\text{const}-\frac{1}{2\sigma^2}\sum_i\big(y_i-f_w(x_i)\big)^2$. Maximizing it over $w$ is exactly minimizing the sum of squared errors. Bernoulli labels give log-loss the same way, and categorical labels give cross-entropy.

## See it

> Figure: Drag μ and σ: the likelihood is the product of the orange bars. ‘Fit by MLE’ moves to μ̂ = x̄ = 6.18, σ̂ = 1.32, where log L peaks at −16.99. The coin tab shows L(p) = pʰ(1 − p)ᵗ peaking at h/n.

## Worked example

**Coin.** 7 heads in 10 flips: $\ell(p)=7\log p+3\log(1-p)$, and $\ell'(p)=\frac7p-\frac{3}{1-p}=0\Rightarrow\hat p=0.7$. Then $\ell(0.7)=-6.109$ versus $\ell(0.5)=-6.931$: the data are $e^{0.823}\approx2.28$ times more likely under $p=0.7$ than under a fair coin.

**Gaussian.** Data $2, 4, 6$: $\hat\mu=4$ and $\hat\sigma^2=\frac{(-2)^2+0^2+2^2}{3}=\frac83\approx2.67$, while the unbiased estimate is $s^2=\frac82=4$.

## In code

```python
import numpy as np
from scipy import stats, optimize

x = np.array([2.0, 4.0, 6.0])
mu_hat, sigma_hat = x.mean(), x.std(ddof=0)       # closed-form MLE: 4.0, 1.633

# The same answer by numerically maximizing the log-likelihood
nll = lambda t: -stats.norm.logpdf(x, t[0], np.exp(t[1])).sum()
res = optimize.minimize(nll, x0=[0.0, 0.0])
res.x[0], np.exp(res.x[1])                         # ≈ 4.0, 1.633

stats.norm.fit(x)                                  # SciPy's MLE: (4.0, 1.633)
```

Closed form, numerical optimization and SciPy's fit agree. Optimizing log σ keeps σ positive.

- **Training ML models:** Minimizing MSE, log-loss or cross-entropy is maximizing a Gaussian, Bernoulli or categorical likelihood.
- **Fitting distributions:** Model waiting times (exponential), counts (Poisson) or durations (log-normal), then simulate or price risk.
- **Model comparison:** AIC $=2k-2\ell(\hat\theta)$ and BIC $=k\log n-2\ell(\hat\theta)$ trade fit against the number of parameters $k$.
- **Latent-variable models:** EM for Gaussian mixtures and hidden Markov models maximizes a likelihood with unobserved labels.

> **MLE overfits small samples:** MLE trusts the data completely: three heads in three flips gives $\hat p=1$ — tails declared impossible. A prior (MAP estimation) or regularization fixes this; a Gaussian prior on weights adds exactly an $L_2$ penalty to $-\ell$.

> **Likelihood is not P(θ | data):** $L(\theta)=p(\text{data}\mid\theta)$, not $p(\theta\mid\text{data})$. Turning it into a distribution over $\theta$ requires a prior and Bayes' theorem.

**Question:** You observe counts $2, 3, 7$ from a Poisson($\lambda$) distribution. The MLE $\hat\lambda$ is…

- A. 3
- B. 4
- C. 7
- D. 12

**Answer:** B. 4 — $\ell(\lambda)=\sum_i(x_i\log\lambda-\lambda)+\text{const}$, so $\ell'(\lambda)=\frac{\sum_i x_i}{\lambda}-n=0\Rightarrow\hat\lambda=\bar x=\frac{12}{3}=4$.

**Question:** Minimizing mean squared error in regression is maximum likelihood when the noise is…

- A. uniform
- B. Gaussian with constant variance
- C. Laplace
- D. Poisson

**Answer:** B. Gaussian with constant variance — The Gaussian log-density is $-\frac{(y-f)^2}{2\sigma^2}+\text{const}$. Laplace noise gives absolute error instead.

**Key points**

- $\hat\theta=\arg\max_\theta\sum_i\log p(x_i\mid\theta)$ for i.i.d. data.
- Gaussian: $\hat\mu=\bar x$, $\hat\sigma^2=\frac1n\sum(x_i-\bar x)^2$ — biased by $\frac{n-1}{n}$.
- Least squares = Gaussian MLE; log-loss = Bernoulli MLE.
- MLE overfits small samples — add a prior (MAP) or regularize.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
