Random Variables & Expectation

Expectation is the probability-weighted average, linear even for dependent variables; variance is the expected squared distance from it.

6 min read347 wordscore3 viewsupdated 2026-10-03

Definition

A random variable XX assigns a number to each outcome of a random experiment. Its expectation E[X]\mathbb{E}[X] is the probability-weighted average of its values; its variance Var⁡(X)=E[(X−E[X])2]\operatorname{Var}(X)=\mathbb{E}\big[(X-\mathbb{E}[X])^2\big] is the expected squared distance from that average.

At a glance

Notation
XX variable, xx value; p(x)=P(X=x)p(x)=P(X=x) (PMF) or f(x)f(x) (PDF); μ=E[X]\mu=\mathbb{E}[X], σ2=Var⁡(X)\sigma^2=\operatorname{Var}(X)
Discrete
E[X]=∑xx p(x)\mathbb{E}[X]=\sum_x x\,p(x)
Continuous
E[X]=∫x f(x) dx\mathbb{E}[X]=\int x\,f(x)\,dx
Units
μ\mu and σ\sigma in the units of XX; σ2\sigma^2 in squared units
Prerequisites
probability, sums, integrals
Used in
loss functions, SGD, risk and pricing, every estimator

The intuition

Place a weight equal to p(x)p(x) at each value xx on a beam. E[X]\mathbb{E}[X] is where the beam balances — the centre of mass. Variance measures how far that mass sits from the balance point, squared; its square root σ\sigma is back in the units of XX and gives the typical size of a deviation.

The expectation need not be a possible value — a fair die has E[X]=3.5\mathbb{E}[X]=3.5. It is a long-run average: by the law of large numbers, the mean of many independent draws approaches it.

The math

E[X]=∑xx p(x),Var⁡(X)=E[(X−μ)2]=E[X2]−μ2\mathbb{E}[X]=\sum_x x\,p(x),\qquad \operatorname{Var}(X)=\mathbb{E}\big[(X-\mu)^2\big]=\mathbb{E}[X^2]-\mu^2
μ\mu
E[X]\mathbb{E}[X]
E[X2]\mathbb{E}[X^2]
∑xx2 p(x)\sum_x x^2\,p(x), the second moment

Linearity holds for any random variables, dependent or not:

E[aX+bY+c]=a E[X]+b E[Y]+c\mathbb{E}[aX+bY+c]=a\,\mathbb{E}[X]+b\,\mathbb{E}[Y]+c

Proof (discrete): E[X+Y]=∑x,y(x+y) p(x,y)=∑xx p(x)+∑yy p(y)\mathbb{E}[X+Y]=\sum_{x,y}(x+y)\,p(x,y)=\sum_x x\,p(x)+\sum_y y\,p(y), because summing the joint distribution over one variable gives the other's marginal. Independence is never used.

Variance is not linear. Shifts drop out, scales come out squared, and sums pick up a covariance term:

Var⁡(aX+b)=a2Var⁡(X),Var⁡(X+Y)=Var⁡(X)+Var⁡(Y)+2Cov⁡(X,Y)\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X),\qquad \operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)

For independent (or merely uncorrelated) variables the covariance is zero, so variances add — standard deviations do not. For a function gg, E[g(X)]=∑xg(x) p(x)\mathbb{E}[g(X)]=\sum_x g(x)\,p(x), but in general E[g(X)]≠g(E[X])\mathbb{E}[g(X)]\ne g(\mathbb{E}[X]); for convex gg, Jensen's inequality gives E[g(X)]≥g(E[X])\mathbb{E}[g(X)]\ge g(\mathbb{E}[X]).

See it

drag a bar · the beam balances at E[X]0.20.4±σ0.1010.1020.1030.1040.1050.506E[X] = 4.500
running average of 0 draws (log scale)110100100013.56draw samples to watch the average settle on E[X]
E[X] = Σ x·p(x) = 4.500Var(X) = 3.250σ = 1.803
fig.Drag a bar and the fulcrum slides to E[X]. The 'loaded' preset (P(6) = 0.5) has E[X] = 4.5 and σ = 1.80. Draw samples and the running average settles onto E[X], as the law of large numbers predicts.

Worked example

A fair die: E[X]=1+⋯+66=3.5\mathbb{E}[X]=\tfrac{1+\cdots+6}{6}=3.5 and E[X2]=916\mathbb{E}[X^2]=\tfrac{91}{6}, so Var⁡(X)=916−3.52=3512≈2.917\operatorname{Var}(X)=\tfrac{91}{6}-3.5^2=\tfrac{35}{12}\approx2.917 and σ≈1.708\sigma\approx1.708.

The sum of two dice has E=3.5+3.5=7\mathbb{E}=3.5+3.5=7 (linearity) and, because the dice are independent, Var⁡=2⋅3512≈5.833\operatorname{Var}=2\cdot\tfrac{35}{12}\approx5.833. Its standard deviation is 5.833≈2.415\sqrt{5.833}\approx2.415 — not 2×1.708=3.4162\times1.708=3.416.

In code

python
1import numpy as np
2
3x = np.arange(1, 7)
4p = np.full(6, 1 / 6) # fair die
5mean = (x * p).sum() # 3.5
6var = ((x - mean) ** 2 * p).sum() # 2.9167 = 35/12
7
8rng = np.random.default_rng(0)
9rolls = rng.choice(x, p=p, size=(100_000, 2))
10rolls.sum(1).mean(), rolls.sum(1).var() # ≈ 7.0, ≈ 5.83
Exact moments from the PMF, then a simulation that agrees.

Where it's used

Loss functions

Training minimizes an expected loss E(x,y)[ℓ(f(x),y)]\mathbb{E}_{(x,y)}[\ell(f(x),y)]; the training-set average estimates it.

Stochastic gradient descent

A mini-batch gradient is unbiased — its expectation equals the full gradient — by linearity alone.

Decisions & pricing

Expected revenue, expected cost of an error, expected value of a bet: choose the action with the best expectation.

Estimator quality

Bias is E[θ^]−θ\mathbb{E}[\hat\theta]-\theta and precision is Var⁡(θ^)\operatorname{Var}(\hat\theta); error bars report σ\sigma.

Check yourself

E[X]=2\mathbb{E}[X]=2 and Var⁡(X)=3\operatorname{Var}(X)=3. What is Var⁡(5−2X)\operatorname{Var}(5-2X)?

Check yourself

XX and YY are strongly dependent, with E[X]=1\mathbb{E}[X]=1 and E[Y]=4\mathbb{E}[Y]=4. Then E[X+Y]\mathbb{E}[X+Y]…

Remember

  • ✓E[X]=∑xx p(x)\mathbb{E}[X]=\sum_x x\,p(x) — the balance point of the distribution.
  • ✓Linearity: E[aX+bY+c]=aE[X]+bE[Y]+c\mathbb{E}[aX+bY+c]=a\mathbb{E}[X]+b\mathbb{E}[Y]+c, always.
  • ✓Var⁡(X)=E[X2]−E[X]2\operatorname{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2; Var⁡(aX+b)=a2Var⁡(X)\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X).
  • ✓Variances of independent variables add; E[g(X)]≠g(E[X])\mathbb{E}[g(X)]\ne g(\mathbb{E}[X]) in general.
← back to Mathematics roadmap