# Random Variables & Expectation

> Expectation is the probability-weighted average, linear even for dependent variables; variance is the expected squared distance from it.

Source: https://www.mlfoundry.in/learn/random-variables-expectation  
Section: Mathematics › Probability  
Level: core · 6 min read · updated 2026-10-03

**A random variable:** $X$ assigns a number to each outcome of a random experiment. Its **expectation** $\mathbb{E}[X]$ is the probability-weighted average of its values; its **variance** $\operatorname{Var}(X)=\mathbb{E}\big[(X-\mathbb{E}[X])^2\big]$ is the expected squared distance from that average.

- **Notation:** $X$ variable, $x$ value; $p(x)=P(X=x)$ (PMF) or $f(x)$ (PDF); $\mu=\mathbb{E}[X]$, $\sigma^2=\operatorname{Var}(X)$
- **Discrete:** $\mathbb{E}[X]=\sum_x x\,p(x)$
- **Continuous:** $\mathbb{E}[X]=\int x\,f(x)\,dx$
- **Units:** $\mu$ and $\sigma$ in the units of $X$; $\sigma^2$ in squared units
- **Prerequisites:** probability, sums, integrals
- **Used in:** loss functions, SGD, risk and pricing, every estimator

## The intuition

Place a weight equal to $p(x)$ at each value $x$ on a beam. $\mathbb{E}[X]$ is where the beam balances — the centre of mass. Variance measures how far that mass sits from the balance point, squared; its square root $\sigma$ is back in the units of $X$ and gives the typical size of a deviation.

The expectation need not be a possible value — a fair die has $\mathbb{E}[X]=3.5$. It is a long-run average: by the law of large numbers, the mean of many independent draws approaches it.

## The math

$$
\mathbb{E}[X]=\sum_x x\,p(x),\qquad \operatorname{Var}(X)=\mathbb{E}\big[(X-\mu)^2\big]=\mathbb{E}[X^2]-\mu^2
$$

- $\mu$: $\mathbb{E}[X]$
- $\mathbb{E}[X^2]$: $\sum_x x^2\,p(x)$, the second moment

**Linearity** holds for any random variables, dependent or not:

$$
\mathbb{E}[aX+bY+c]=a\,\mathbb{E}[X]+b\,\mathbb{E}[Y]+c
$$

Proof (discrete): $\mathbb{E}[X+Y]=\sum_{x,y}(x+y)\,p(x,y)=\sum_x x\,p(x)+\sum_y y\,p(y)$, because summing the joint distribution over one variable gives the other's marginal. Independence is never used.

Variance is **not** linear. Shifts drop out, scales come out squared, and sums pick up a covariance term:

$$
\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X),\qquad \operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)
$$

For independent (or merely uncorrelated) variables the covariance is zero, so variances add — standard deviations do not. For a function $g$, $\mathbb{E}[g(X)]=\sum_x g(x)\,p(x)$, but in general $\mathbb{E}[g(X)]\ne g(\mathbb{E}[X])$; for convex $g$, **Jensen's inequality** gives $\mathbb{E}[g(X)]\ge g(\mathbb{E}[X])$.

## See it

> Figure: Drag a bar and the fulcrum slides to E[X]. The 'loaded' preset (P(6) = 0.5) has E[X] = 4.5 and σ = 1.80. Draw samples and the running average settles onto E[X], as the law of large numbers predicts.

## Worked example

A fair die: $\mathbb{E}[X]=\tfrac{1+\cdots+6}{6}=3.5$ and $\mathbb{E}[X^2]=\tfrac{91}{6}$, so $\operatorname{Var}(X)=\tfrac{91}{6}-3.5^2=\tfrac{35}{12}\approx2.917$ and $\sigma\approx1.708$.

The sum of two dice has $\mathbb{E}=3.5+3.5=7$ (linearity) and, because the dice are independent, $\operatorname{Var}=2\cdot\tfrac{35}{12}\approx5.833$. Its standard deviation is $\sqrt{5.833}\approx2.415$ — not $2\times1.708=3.416$.

## In code

```python
import numpy as np

x = np.arange(1, 7)
p = np.full(6, 1 / 6)                       # fair die
mean = (x * p).sum()                        # 3.5
var = ((x - mean) ** 2 * p).sum()           # 2.9167 = 35/12

rng = np.random.default_rng(0)
rolls = rng.choice(x, p=p, size=(100_000, 2))
rolls.sum(1).mean(), rolls.sum(1).var()     # ≈ 7.0, ≈ 5.83
```

Exact moments from the PMF, then a simulation that agrees.

- **Loss functions:** Training minimizes an *expected* loss $\mathbb{E}_{(x,y)}[\ell(f(x),y)]$; the training-set average estimates it.
- **Stochastic gradient descent:** A mini-batch gradient is unbiased — its expectation equals the full gradient — by linearity alone.
- **Decisions & pricing:** Expected revenue, expected cost of an error, expected value of a bet: choose the action with the best expectation.
- **Estimator quality:** Bias is $\mathbb{E}[\hat\theta]-\theta$ and precision is $\operatorname{Var}(\hat\theta)$; error bars report $\sigma$.

> **E[g(X)] ≠ g(E[X]):** The mean of squares is not the square of the mean, and the expected log is not the log of the expectation. Applying a nonlinear function after averaging — e.g. averaging returns instead of log-returns — silently biases results.

> **Standard deviations don't add:** For independent $X$ and $Y$, $\sigma_{X+Y}=\sqrt{\sigma_X^2+\sigma_Y^2}$, not $\sigma_X+\sigma_Y$. Independent errors combine in quadrature.

> **The mean minimizes squared error:** $\mathbb{E}[(X-c)^2]=\operatorname{Var}(X)+(\mathbb{E}[X]-c)^2$ is smallest at $c=\mathbb{E}[X]$. That is why MSE regression predicts conditional means — and why the median, which minimizes absolute error, is the robust alternative.

**Question:** $\mathbb{E}[X]=2$ and $\operatorname{Var}(X)=3$. What is $\operatorname{Var}(5-2X)$?

- A. −3
- B. 3
- C. 12
- D. −6

**Answer:** C. 12 — $\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X)=(-2)^2\cdot3=12$. The shift by 5 has no effect, and a variance is never negative.

**Question:** $X$ and $Y$ are strongly dependent, with $\mathbb{E}[X]=1$ and $\mathbb{E}[Y]=4$. Then $\mathbb{E}[X+Y]$…

- A. = 5
- B. < 5
- C. > 5
- D. cannot be found without the joint distribution

**Answer:** A. = 5 — Linearity of expectation needs no independence: E[X + Y] = E[X] + E[Y] = 5.

**Key points**

- $\mathbb{E}[X]=\sum_x x\,p(x)$ — the balance point of the distribution.
- Linearity: $\mathbb{E}[aX+bY+c]=a\mathbb{E}[X]+b\mathbb{E}[Y]+c$, always.
- $\operatorname{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2$; $\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X)$.
- Variances of independent variables add; $\mathbb{E}[g(X)]\ne g(\mathbb{E}[X])$ in general.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
