# Entropy

> H(p) = −Σ p log p is the average surprise of a distribution — the minimum average bits per symbol — maximal when p is uniform.

Source: https://www.mlfoundry.in/learn/entropy  
Section: Mathematics › Information Theory  
Level: core · 5 min read · updated 2026-10-03

**Entropy:** $H(X)=-\sum_x p(x)\log p(x)$ is the expected **surprise** of a random variable $X$ with distribution $p$: how uncertain its outcome is before you observe it, measured as the average number of bits an optimal code needs per outcome.

- **Notation:** $H(X)$ or $H(p)$; convention $0\log 0=0$
- **Units:** bits with $\log_2$, nats with $\ln$; $1\text{ nat}=1/\ln2\approx1.443$ bits
- **Range:** $0\le H(X)\le\log_2 K$ for $K$ outcomes
- **Key property:** zero iff the outcome is certain; maximal iff all outcomes are equally likely
- **Prerequisites:** random variables & expectation, logarithms
- **Used in:** decision-tree splits, cross-entropy loss, mutual information, compression, perplexity

## The intuition

Learning that a certain event happened tells you nothing. Learning that a one-in-a-million event happened tells you a lot. So the information in an outcome should grow as its probability falls — and for two independent outcomes, the information should add. The logarithm is the function that does both: $p(x)p(y)$ becomes $-\log p(x)-\log p(y)$.

Entropy is that surprise, averaged over outcomes. A fair coin has 1 bit: one yes/no question settles it. A coin that lands heads 90% of the time has only 0.469 bits — you mostly know what is coming.

## The math

$$
I(x)=-\log_2 p(x),\qquad H(X)=\mathbb{E}\big[I(X)\big]=-\sum_{x}p(x)\log_2 p(x)
$$

- $I(x)$: surprise (self-information) of outcome $x$, in bits
- $p(x)$: probability of outcome $x$

**Bounds.** Each term is non-negative, so $H\ge0$, with equality only when one outcome has probability 1. By Jensen's inequality, $H(X)=\mathbb{E}[\log_2\tfrac{1}{p(X)}]\le\log_2\mathbb{E}[\tfrac{1}{p(X)}]=\log_2K$, with equality only for the uniform distribution.

**Coding.** Shannon's source-coding theorem: no uniquely decodable code can use fewer than $H$ bits per symbol on average, and a Huffman code always gets within one bit:

$$
H(X)\ \le\ L_{\text{Huffman}}\ <\ H(X)+1,\qquad L=\sum_x p(x)\,\ell(x)
$$

- $\ell(x)$: length in bits of the codeword for $x$

**Two variables.** The chain rule $H(X,Y)=H(X)+H(Y\mid X)$ splits joint uncertainty into "what $X$ tells you" plus "what remains about $Y$". Conditioning never increases entropy on average, and the reduction is the **mutual information**:

$$
I(X;Y)=H(Y)-H(Y\mid X)\ \ge 0,\qquad I(X;Y)=0\iff X\perp Y
$$

## See it

> Figure: Drag the bars. Orange labels are each symbol's surprise. The skewed preset (½, ¼, ⅛, ⅛) has H = 1.75 bits, and its Huffman code (1, 2, 3, 3 bits) meets it exactly; with 2 symbols the inset traces H(p), peaking at 1 bit.

## Worked example

$p=(\tfrac12,\tfrac14,\tfrac18,\tfrac18)$: $H=\tfrac12\cdot1+\tfrac14\cdot2+\tfrac18\cdot3+\tfrac18\cdot3=1.75$ bits. The codewords 0, 10, 110, 111 have lengths equal to each symbol's surprise, so $L=1.75$ — optimal. Uniform over 4 outcomes needs $\log_24=2$ bits; skew saves 0.25 bits per symbol.

**Information gain.** A node with 5 positives and 5 negatives has $H=1$ bit. A split sends (4+, 0−) left and (1+, 5−) right: $H_L=0$, $H_R=-\tfrac16\log_2\tfrac16-\tfrac56\log_2\tfrac56=0.650$. Weighted child entropy $=0.4\cdot0+0.6\cdot0.650=0.390$, so the gain is $1-0.390=0.610$ bits.

## In code

```python
import numpy as np
from scipy.stats import entropy

p = np.array([0.5, 0.25, 0.125, 0.125])
-(p * np.log2(p)).sum()            # 1.75 bits
entropy(p, base=2)                 # 1.75 bits
entropy(p)                         # 1.213 nats

def info_gain(y, mask):            # y: integer labels, mask: boolean split
    H = lambda v: entropy(np.bincount(v), base=2)
    w = mask.mean()
    return H(y) - w * H(y[mask]) - (1 - w) * H(y[~mask])
```

scipy.stats.entropy normalizes its input, handles zeros, and uses nats unless you pass base=2.

- **Decision trees:** ID3 / C4.5 and `criterion="entropy"` choose the split with the largest information gain.
- **Feature selection:** Mutual information $I(X;Y)$ ranks features by how much they reduce uncertainty about the target — and, unlike correlation, catches nonlinear dependence.
- **Uncertainty sampling:** Active learning labels the examples whose predicted class distribution has the highest entropy.
- **Reinforcement learning:** An entropy bonus on the policy keeps it exploring instead of collapsing onto one action too early.

> **Plug-in estimates are biased low:** Estimating $H$ by plugging sample frequencies into the formula underestimates it — outcomes you haven't seen yet contribute nothing. With few samples per category (small $n$, large $K$), the bias is large; mutual-information feature scores inherit it.

> **Differential entropy is a different animal:** For a density, $h=-\int f\log f$ can be negative and changes when you rescale the variable (a Gaussian has $h=\tfrac12\log(2\pi e\sigma^2)$). Don't compare it with discrete entropy; differences of it, such as KL and mutual information, are the meaningful quantities.

> **Only probabilities matter:** Entropy ignores what the outcomes are. Values {1, 1000, 10⁶} with equal probabilities have the same entropy as {A, B, C}. It measures uncertainty about *which* outcome — not spread, which is what variance measures.

**Question:** A fair 8-sided die has entropy…

- A. 1 bit
- B. 2 bits
- C. 3 bits
- D. 8 bits

**Answer:** C. 3 bits — Uniform over $K=8$ outcomes: $H=\log_2 8=3$ bits — three yes/no questions identify the face.

**Question:** Which distribution over 4 outcomes has the highest entropy?

- A. (1, 0, 0, 0)
- B. (0.5, 0.5, 0, 0)
- C. (0.4, 0.3, 0.2, 0.1)
- D. (0.25, 0.25, 0.25, 0.25)

**Answer:** D. (0.25, 0.25, 0.25, 0.25) — The uniform distribution attains the maximum $\log_24=2$ bits. The others give 0, 1 and 1.846 bits.

**Key points**

- Surprise $-\log p(x)$; entropy is its average, $-\sum p\log p$.
- $0\le H\le\log_2K$: zero when certain, maximal when uniform.
- $H$ is the floor on average code length; Huffman achieves $H\le L<H+1$.
- Information gain and mutual information are reductions in entropy.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
