# Covariance & Correlation

> Covariance measures how two variables move together; correlation rescales it to [−1, 1] and captures only linear association.

Source: https://www.mlfoundry.in/learn/covariance-correlation  
Section: Mathematics › Probability  
Level: core · 5 min read · updated 2026-10-03

**Covariance:** $\operatorname{Cov}(X,Y)=\mathbb{E}\big[(X-\mu_X)(Y-\mu_Y)\big]$ measures whether two variables tend to sit above their means together (positive) or on opposite sides (negative). The **Pearson correlation** $\rho=\operatorname{Cov}(X,Y)/(\sigma_X\sigma_Y)$ rescales it to $[-1,1]$, free of units.

- **Notation:** $\operatorname{Cov}(X,Y)$ or $\sigma_{XY}$; $\rho_{XY}$ (population), $r$ (sample)
- **Range:** covariance: any real number, in units of $X$ × units of $Y$; correlation: $[-1,1]$, unitless
- **ρ = ±1:** exactly when $Y=aX+b$ with $a\ne0$; the sign of $a$ gives the sign of $\rho$
- **ρ = 0:** no *linear* association — dependence can remain
- **Covariance matrix:** $\Sigma=\frac{1}{n-1}X_c^\top X_c$ for centred data $X_c$; symmetric, positive semi-definite
- **Used in:** PCA, Gaussian models, portfolio risk, feature screening

## The intuition

Centre both variables at their means. Every point now lies in one of four quadrants. Points top-right or bottom-left — $x$ and $y$ deviating the same way — contribute a positive product $(x-\bar x)(y-\bar y)$; points in the other two quadrants contribute negative products. Covariance is the average of these signed rectangle areas: which way the cloud tilts, weighted by how far points stray.

Covariance depends on units — measure height in centimetres instead of metres and it grows 100×. Dividing by both standard deviations removes the units and leaves a pure number, the correlation, which is simply the covariance of the standardized variables $z_x=(x-\bar x)/s_x$ and $z_y=(y-\bar y)/s_y$.

## The math

$$
\operatorname{Cov}(X,Y)=\mathbb{E}[XY]-\mu_X\mu_Y,\qquad r=\frac{\sum_i(x_i-\bar x)(y_i-\bar y)}{\sqrt{\sum_i(x_i-\bar x)^2\,\sum_i(y_i-\bar y)^2}}
$$

- $\bar x,\ \bar y$: sample means
- $r$: sample (Pearson) correlation

$\operatorname{Cov}(X,X)=\operatorname{Var}(X)$, and covariance is bilinear: $\operatorname{Cov}(aX+b,\,cY+d)=ac\operatorname{Cov}(X,Y)$. So correlation is unchanged by shifting either variable or rescaling it by a positive factor. By the Cauchy–Schwarz inequality $|\operatorname{Cov}(X,Y)|\le\sigma_X\sigma_Y$, which is why $|\rho|\le1$.

Correlation is regression in standardized units: the least-squares line of $y$ on $x$ has slope $b=r\,s_y/s_x$, and $r^2$ is the fraction of the variance of $y$ explained by that line. For a random vector, all pairwise covariances form the covariance matrix $\Sigma$, and the variance of any linear combination follows from it:

$$
\operatorname{Var}(w^\top X)=w^\top\Sigma\,w
$$

## See it

> Figure: Points in the green quadrants add to the covariance, red ones subtract. Set ρ and the sample r matches it exactly. Under 'where r misleads', a parabola and a circle give r ≈ 0 despite perfect dependence, and one outlier turns r = −0.88 into +0.67.

## Worked example

Hours studied $x=(1,2,3,4,5)$, exam score $y=(52,55,61,64,68)$; means $3$ and $60$. Deviations $x-\bar x=(-2,-1,0,1,2)$ and $y-\bar y=(-8,-5,1,4,8)$. Products sum to $16+5+0+4+16=41$, so $s_{xy}=41/4=10.25$.

With $s_x=\sqrt{10/4}=1.581$ and $s_y=\sqrt{170/4}=6.519$: $r=\frac{10.25}{1.581\cdot6.519}=0.994$. The fitted slope is $r\,s_y/s_x=41/10=4.1$ points per hour.

## In code

```python
import numpy as np
from scipy import stats

x = np.array([1, 2, 3, 4, 5])
y = np.array([52, 55, 61, 64, 68])
np.cov(x, y)               # [[2.5, 10.25], [10.25, 42.5]]
np.corrcoef(x, y)[0, 1]    # 0.9944
stats.spearmanr(x, y)      # statistic = 1.0: y rises monotonically with x
```

np.cov uses the n − 1 denominator by default; Spearman correlates ranks.

- **Feature screening:** A correlation heatmap flags redundant features (|r| > 0.9) that destabilize linear-model weights.
- **PCA:** Principal components are the eigenvectors of the covariance matrix.
- **Finance:** Portfolio variance is $w^\top\Sigma w$; diversification works because correlations are below 1.
- **Gaussian models:** A multivariate Gaussian is defined by a mean vector and a covariance matrix; GMMs and Kalman filters estimate Σ.

> **Correlation is not causation:** Ice-cream sales and drownings are correlated because both rise in summer. Correlation measures association; establishing cause needs an experiment or a causal model that rules out confounders.

> **Uncorrelated is not independent:** If $X$ is symmetric about 0 and $Y=X^2$, then $\operatorname{Cov}(X,Y)=\mathbb{E}[X^3]-\mathbb{E}[X]\,\mathbb{E}[X^2]=0$, yet $Y$ is a function of $X$. Independence implies zero correlation; the converse holds only in special cases, such as jointly Gaussian variables.

> **Robust alternatives:** Pearson's r is pulled around by outliers and blind to curvature. Spearman's rank correlation — Pearson on the ranks — detects any monotonic relationship and resists outliers. Always plot the data.

**Question:** Temperatures are converted from °C to °F (F = 1.8·C + 32). The correlation between temperature and ice-cream sales…

- A. is multiplied by 1.8
- B. increases by 32
- C. is unchanged
- D. changes sign

**Answer:** C. is unchanged — $\operatorname{Cov}(1.8C+32,\,S)=1.8\operatorname{Cov}(C,S)$ and $\sigma_F=1.8\,\sigma_C$; the factors cancel in $\rho$.

**Question:** $X\sim\text{Uniform}(-1,1)$ and $Y=X^2$. The correlation between $X$ and $Y$ is…

- A. 1
- B. about 0.5
- C. 0
- D. −1

**Answer:** C. 0 — $\operatorname{Cov}(X,X^2)=\mathbb{E}[X^3]-\mathbb{E}[X]\,\mathbb{E}[X^2]=0-0=0$, although $Y$ is completely determined by $X$.

**Key points**

- $\operatorname{Cov}(X,Y)=\mathbb{E}[(X-\mu_X)(Y-\mu_Y)]$: the average signed product of deviations.
- $\rho=\operatorname{Cov}/(\sigma_X\sigma_Y)\in[-1,1]$, unit-free; $\pm1$ only for exact linear relations.
- $r=0$ means no linear association — not independence.
- Correlation is not causation; plot the data, check outliers, consider Spearman.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
