Definition
Entropy is the expected surprise of a random variable with distribution : how uncertain its outcome is before you observe it, measured as the average number of bits an optimal code needs per outcome.
At a glance
- Notation
- or ; convention
- Units
- bits with , nats with ; bits
- Range
- for outcomes
- Key property
- zero iff the outcome is certain; maximal iff all outcomes are equally likely
- Prerequisites
- random variables & expectation, logarithms
- Used in
- decision-tree splits, cross-entropy loss, mutual information, compression, perplexity
The intuition
Learning that a certain event happened tells you nothing. Learning that a one-in-a-million event happened tells you a lot. So the information in an outcome should grow as its probability falls — and for two independent outcomes, the information should add. The logarithm is the function that does both: becomes .
Entropy is that surprise, averaged over outcomes. A fair coin has 1 bit: one yes/no question settles it. A coin that lands heads 90% of the time has only 0.469 bits — you mostly know what is coming.
The math
- surprise (self-information) of outcome , in bits
- probability of outcome
Bounds. Each term is non-negative, so , with equality only when one outcome has probability 1. By Jensen's inequality, , with equality only for the uniform distribution.
Coding. Shannon's source-coding theorem: no uniquely decodable code can use fewer than bits per symbol on average, and a Huffman code always gets within one bit:
- length in bits of the codeword for
Two variables. The chain rule splits joint uncertainty into "what tells you" plus "what remains about ". Conditioning never increases entropy on average, and the reduction is the mutual information:
See it
Huffman lengths: A1B2C3D3
avg code length L = 1.750 · H ≤ L < H + 1 ✓
Set symbols to 2 to see the binary entropy curve.
Worked example
: bits. The codewords 0, 10, 110, 111 have lengths equal to each symbol's surprise, so — optimal. Uniform over 4 outcomes needs bits; skew saves 0.25 bits per symbol.
Information gain. A node with 5 positives and 5 negatives has bit. A split sends (4+, 0−) left and (1+, 5−) right: , . Weighted child entropy , so the gain is bits.
In code
1import numpy as np2from scipy.stats import entropy34p = np.array([0.5, 0.25, 0.125, 0.125])5-(p * np.log2(p)).sum() # 1.75 bits6entropy(p, base=2) # 1.75 bits7entropy(p) # 1.213 nats89def info_gain(y, mask): # y: integer labels, mask: boolean split10 H = lambda v: entropy(np.bincount(v), base=2)11 w = mask.mean()12 return H(y) - w * H(y[mask]) - (1 - w) * H(y[~mask])Where it's used
Decision trees
ID3 / C4.5 and criterion="entropy" choose the split with the largest information gain.
Feature selection
Mutual information ranks features by how much they reduce uncertainty about the target — and, unlike correlation, catches nonlinear dependence.
Uncertainty sampling
Active learning labels the examples whose predicted class distribution has the highest entropy.
Reinforcement learning
An entropy bonus on the policy keeps it exploring instead of collapsing onto one action too early.
Check yourself
A fair 8-sided die has entropy…
Check yourself
Which distribution over 4 outcomes has the highest entropy?
Remember
- ✓Surprise ; entropy is its average, .
- ✓: zero when certain, maximal when uniform.
- ✓ is the floor on average code length; Huffman achieves .
- ✓Information gain and mutual information are reductions in entropy.