# Factor Analysis

> Model each feature as a linear combination of a few latent factors plus its own independent noise.

Source: https://www.mlfoundry.in/learn/factor-analysis  
Section: ML Algorithms › Unsupervised · Dimensionality Reduction  
Level: advanced · 19 min read · updated 2026-10-08

**Factor analysis (FA):** is a latent-variable model that explains the correlations among $p$ observed variables with a small number $k$ of unobserved **factors**. Each variable is a weighted sum of the factors plus a noise term of its own, and the noise terms are independent of each other. All the covariance *between* variables is therefore carried by the factors.

- **Task:** Unsupervised: latent structure, dimensionality reduction, covariance modelling
- **Input → output:** $X\in\mathbb{R}^{n\times p}$ → loadings $\Lambda\in\mathbb{R}^{p\times k}$, uniquenesses $\psi_1,\dots,\psi_p$, and factor scores $\hat z\in\mathbb{R}^{k}$ per row
- **Learns:** mean $\mu$, loading matrix $\Lambda$, diagonal noise covariance $\Psi$, by maximum likelihood
- **Training:** EM (or direct likelihood optimization); each iteration costs $O(p^2k)$ once the $p\times p$ covariance has been formed in $O(np^2)$
- **Prediction:** factor scores $\hat z=\beta(x-\mu)$, which costs $O(pk)$
- **Hyperparameters:** number of factors $k$, rotation (none, varimax, quartimax, oblique), whether to work on correlations or covariances
- **Assumes:** linear effects of the factors, Gaussian factors and noise, noise independent across variables

## The problem

A class sits four tests: Math, Physics, Literature and Chemistry. The scores are all positively correlated. Students who do well in Math tend to do well in Chemistry ($r=0.81$), and even Literature correlates 0.56–0.59 with the others. One natural explanation is that a single hidden trait, call it *general academic ability*, pushes all four scores up or down together, and that each test then adds its own luck and test-specific skill on top.

We cannot measure that trait directly. Factor analysis asks whether such a trait is *consistent with the data*, how strongly each test reflects it, and how much of each test is just its own noise. The same question comes up whenever many measurements look like noisy readings of a few underlying quantities: survey items and attitudes, stock returns and market sectors, gene expression and pathways.

## The idea

Imagine a generative story. First draw each student's factor values $z$, such as ability. Then produce each test score as a weighted sum of those factors plus independent noise. Because every test shares the same $z$, the tests end up correlated. Because the noise is private to each test, it adds variance to that test but no covariance with any other test.

Fitting the model runs that story backwards. We look for the weights (**loadings**) and noise levels that make the model's implied covariance matrix as close as possible to the observed one. The split is the key output: each variable's variance is divided into a **shared** part, which the factors explain, and a **unique** part, which they do not.

**Vocabulary**

- **Factor (latent variable):** an unobserved quantity $z_m$; by convention $z\sim\mathcal N(0,I)$, so factors have unit variance and are uncorrelated
- **Loading:** $\lambda_{jm}$, the weight of factor $m$ in variable $j$; on standardized data it is the correlation between the variable and the factor
- **Communality:** $h_j^2=\sum_m\lambda_{jm}^2$, the part of variable $j$'s variance that the factors explain
- **Uniqueness:** $\psi_j$, the variance of variable $j$'s own noise (specific factor plus measurement error)
- **Factor score:** an estimate of a row's $z$, usually the posterior mean $\mathbb E[z\mid x]$
- **Rotation:** replacing $\Lambda$ by $\Lambda Q$ for an orthogonal $Q$; the fit does not change, but the loadings become easier to interpret

## The model

$$
x=\mu+\Lambda z+\varepsilon,\qquad z\sim\mathcal N(0,I_k),\qquad \varepsilon\sim\mathcal N(0,\Psi),\quad \Psi=\operatorname{diag}(\psi_1,\dots,\psi_p)
$$

- $x\in\mathbb R^p$: one observed row, for example one student's four test scores
- $\mu$: mean of each variable
- $\Lambda\in\mathbb R^{p\times k}$: factor loading matrix, one row per variable and one column per factor
- $z\in\mathbb R^k$: latent factors, independent of $\varepsilon$, with $k\ll p$
- $\varepsilon$: unique noise; $\Psi$ is **diagonal**, so the noise terms are independent across variables

Since $x$ is a linear function of Gaussians, it is Gaussian too. Its covariance is the sum of a low-rank part that comes from the factors and a diagonal part that comes from the noise:

$$
\Sigma=\operatorname{Cov}(x)=\Lambda\Lambda^\top+\Psi,\qquad x\sim\mathcal N(\mu,\;\Lambda\Lambda^\top+\Psi)
$$

- $\Lambda\Lambda^\top$: rank-$k$ shared covariance; its off-diagonal entries are the **only** source of correlation between variables
- $\Psi$: diagonal; it adds to the variances and never to the covariances

$$
\Sigma_{jj}=\underbrace{\textstyle\sum_m\lambda_{jm}^2}_{h_j^2\ \text{(communality)}}+\underbrace{\psi_j}_{\text{uniqueness}},\qquad \Sigma_{ij}=\textstyle\sum_m\lambda_{im}\lambda_{jm}\quad(i\neq j)
$$

- $h_j^2$: variance of variable j explained by the common factors
- $\psi_j$: variance left over; on standardized data $h_j^2+\psi_j=1$

With one factor the off-diagonal structure is very rigid: $\operatorname{Corr}(x_i,x_j)=\lambda_i\lambda_j$. This is a testable claim. A covariance matrix that fits it is one where every correlation is the product of two per-variable numbers. Counting parameters makes the same point: $\Sigma$ has $p(p+1)/2$ free entries, and the model uses $pk+p-k(k-1)/2$ of them, which leaves $\tfrac12[(p-k)^2-(p+k)]$ degrees of freedom. For 4 tests and 1 factor that is 2, so the model genuinely constrains the data.

## Fitting by EM

The log-likelihood of centred data with sample covariance $S$ is $\ell=-\tfrac n2\big[p\log 2\pi+\log|\Sigma|+\operatorname{tr}(\Sigma^{-1}S)\big]$ with $\Sigma=\Lambda\Lambda^\top+\Psi$. There is no closed form for its maximum. If we *knew* each row's $z$, though, the problem would reduce to $p$ ordinary linear regressions of $x_j$ on $z$. **Expectation–maximization** exploits this: it alternates between inferring $z$ given the current parameters and refitting the regressions using those inferred values.

**E-step.** Under the current $\Lambda,\Psi$ the pair $(z,x)$ is jointly Gaussian, so the posterior of $z$ given $x$ is Gaussian with:

$$
\beta=\Lambda^\top(\Lambda\Lambda^\top+\Psi)^{-1}=(I+\Lambda^\top\Psi^{-1}\Lambda)^{-1}\Lambda^\top\Psi^{-1}
$$

- $\beta\in\mathbb R^{k\times p}$: regression of $z$ on $x$. The second form uses the Woodbury identity and only inverts a $k\times k$ matrix plus the diagonal $\Psi$

$$
\mathbb E[z\mid x]=\beta(x-\mu),\qquad \mathbb E[zz^\top\mid x]=\underbrace{I-\beta\Lambda}_{\operatorname{Cov}(z\mid x)}+\beta(x-\mu)(x-\mu)^\top\beta^\top
$$

- $\mathbb E[z\mid x]$: posterior mean of the factors. This is also the regression (Thomson) factor score
- $I-\beta\Lambda$: posterior covariance. It is the same for every row and shrinks as the loadings grow

**M-step.** Maximize the expected complete-data log-likelihood. This is the least-squares regression of $x$ on $z$, with $zz^\top$ replaced by its expectation. Averaged over rows it depends on the data only through $S$:

$$
\Lambda_{\text{new}}=S\beta^\top\big(I-\beta\Lambda+\beta S\beta^\top\big)^{-1},\qquad \Psi_{\text{new}}=\operatorname{diag}\!\big(S-\Lambda_{\text{new}}\,\beta S\big)
$$

- $S\beta^\top$: $\tfrac1n\sum_i (x_i-\mu)\,\mathbb E[z\mid x_i]^\top$, the cross-covariance of the data and the inferred factors
- $I-\beta\Lambda+\beta S\beta^\top$: $\tfrac1n\sum_i\mathbb E[zz^\top\mid x_i]$, the expected second moment of the factors
- $\operatorname{diag}(\cdot)$: keep only the diagonal. This is where the assumption that $\Psi$ is diagonal enters

Each iteration is guaranteed not to decrease the likelihood. Convergence is usually linear and can be slow when a uniqueness heads towards 0. Production code such as scikit-learn's `FactorAnalysis` uses an equivalent SVD-based update, and statistics packages often maximize $\ell$ directly with a quasi-Newton method. The fitted model is the same either way.

**EM for factor analysis**

1. Centre the data (standardize it too if you want to work with correlations) and form $S$.
2. Initialize $\Lambda$ (for example from the top-$k$ PCA loadings, or a constant) and $\Psi=\operatorname{diag}(S)-\operatorname{diag}(\Lambda\Lambda^\top)$, keeping every $\psi_j>0$.
3. **E-step:** $\beta=(I+\Lambda^\top\Psi^{-1}\Lambda)^{-1}\Lambda^\top\Psi^{-1}$.
4. **M-step:** $\Lambda\leftarrow S\beta^\top(I-\beta\Lambda+\beta S\beta^\top)^{-1}$, then $\Psi\leftarrow\operatorname{diag}(S-\Lambda\beta S)$.
5. Repeat until the loadings (or $\ell$) stop changing. Then rotate if you want interpretable factors, and score rows with $\hat z=\beta(x-\mu)$.

## Worked example: one factor behind four tests

Ten students and four tests. We standardize each column (population standard deviation), so $S$ is the correlation matrix $R$ and the loadings are correlations with the factor.

Raw scores. Column means 68.0, 64.6, 72.8, 60.0; standard deviations 10.95, 9.59, 12.00, 12.78.

| Student | Math | Physics | Literature | Chemistry |
| --- | --- | --- | --- | --- |
| 1 | 73 | 62 | 83 | 72 |
| 2 | 75 | 71 | 79 | 60 |
| 3 | 73 | 66 | 70 | 48 |
| 4 | 70 | 69 | 67 | 71 |
| 5 | 82 | 83 | 77 | 81 |
| 6 | 79 | 75 | 92 | 71 |
| 7 | 71 | 52 | 63 | 59 |
| 8 | 51 | 52 | 47 | 40 |
| 9 | 49 | 58 | 68 | 46 |
| 10 | 57 | 58 | 82 | 52 |

Sample correlation matrix R. Literature is the least correlated with the rest.

| r | Math | Physics | Literature | Chemistry |
| --- | --- | --- | --- | --- |
| **Math** | 1 | 0.756 | 0.562 | 0.806 |
| **Physics** | 0.756 | 1 | 0.591 | 0.743 |
| **Literature** | 0.562 | 0.591 | 1 | 0.594 |
| **Chemistry** | 0.806 | 0.743 | 0.594 | 1 |

**The first EM iteration by hand**

1. Start with every $\lambda_j=0.6$ and $\psi_j=0.64$, so each variance is $0.36+0.64=1$. The model correlation is $0.6^2=0.36$ everywhere, far below the observed values. Average log-likelihood: $-4.861$.
2. **E-step.** $w_j=\lambda_j/\psi_j=0.9375$, and $1+\sum_j\lambda_jw_j=1+4(0.6)(0.9375)=3.25$, so $\beta_j=0.9375/3.25=0.2885$ for every test. The posterior variance of a student's factor is $1-\beta\Lambda=1-4(0.2885)(0.6)=0.308$.
3. **M-step, numerator.** $(S\beta^\top)_j=0.2885\times(\text{row sum of }R)$. For Math that is $0.2885\times3.123=0.901$.
4. **M-step, denominator.** $1-\beta\Lambda+\beta R\beta^\top=0.308+0.2885^2\times(\text{sum of all entries of }R=12.10)=0.308+1.007=1.315$.
5. New loadings $\lambda=(0.685,\;0.678,\;0.603,\;0.690)$; new uniquenesses $\psi=(0.383,\;0.396,\;0.523,\;0.375)$. Log-likelihood rises to $-4.565$.
6. Iterate. After 20 rounds every loading is within 0.007 of its final value; after **64** rounds the largest change is below $10^{-6}$ and $\ell=-4.413$.

Converged one-factor solution. Communality + uniqueness = 1 for each test. The factor explains 68.6% of the total variance (2.745 of 4).

| Test | Loading λ | Communality h² = λ² | Uniqueness ψ | Score weight w = λ/ψ |
| --- | --- | --- | --- | --- |
| Math | 0.894 | 0.799 | 0.201 | 4.46 |
| Physics | 0.843 | 0.711 | 0.289 | 2.91 |
| Literature | 0.659 | 0.434 | **0.566** | 1.17 |
| Chemistry | 0.895 | 0.801 | 0.199 | 4.49 |

Literature is the outlier: more than half its variance (0.566) is unique, meaning it is driven by something the other three tests do not share. Math and Chemistry are the cleanest measures of the common factor.

Observed vs reconstructed correlations, $\hat\Sigma_{ij}=\lambda_i\lambda_j$. The diagonal is reproduced exactly ($h^2+\psi=1$); every off-diagonal residual is within ±0.035.

| Pair | Observed r | Model λᵢλⱼ | Residual |
| --- | --- | --- | --- |
| Math · Physics | 0.756 | 0.754 | +0.002 |
| Math · Literature | 0.562 | 0.589 | −0.028 |
| Math · Chemistry | 0.806 | 0.800 | +0.006 |
| Physics · Literature | 0.591 | 0.556 | **+0.035** |
| Physics · Chemistry | 0.743 | 0.754 | −0.011 |
| Literature · Chemistry | 0.594 | 0.590 | +0.005 |

**Scoring a new student.** Take scores (80, 70, 60, 75). As z-scores that is $(1.095,\;0.563,\;-1.067,\;1.174)$. The factor score is $\hat z=\sum_j w_jx_j\,/\,(1+\sum_j\lambda_jw_j)=10.55/12.23=$ **0.863**, with posterior standard deviation $\sqrt{1/12.23}=0.286$. The weak Literature mark contributes only $-1.24$ against $+4.88$ (Math) and $+5.27$ (Chemistry), because Literature is mostly noise as far as the factor is concerned. FA weights each variable by its *reliability*, $\lambda_j/\psi_j$.

> Figure: Tab 1 runs EM on the four-test example. The bars split each test's unit variance into communality λ² and uniqueness ψ, the heatmap shows the residual R − ΛΛᵀ − Ψ shrinking from about 0.4 to within ±0.035, and the log-likelihood climbs from −4.861 to −4.413 in 64 iterations. The last frame scores the new student (ẑ = 0.863). Tab 2 fits two factors to six tests. Drag the F1 axis to rotate the solution: the loadings change, but the communalities and the fit do not. Varimax at −36.3° gives a clean verbal factor and a clean maths factor.

## Rotational indeterminacy and varimax

With $k\ge2$ the loadings are not unique. For any orthogonal $k\times k$ matrix $Q$, setting $\tilde\Lambda=\Lambda Q$ and $\tilde z=Q^\top z$ gives exactly the same distribution for $x$:

$$
\tilde\Lambda\tilde\Lambda^\top=\Lambda QQ^\top\Lambda^\top=\Lambda\Lambda^\top\;\Rightarrow\;\tilde\Sigma=\Sigma
$$

- $Q$: any rotation or reflection, $QQ^\top=I$
- $h_j^2$: row norms of $\Lambda$; these are unchanged by $Q$, so communalities and uniquenesses are identified while individual loadings are not

The likelihood cannot choose among these solutions, so we choose one that is easy to *interpret*. The aim is **simple structure**: each variable loads strongly on one factor and weakly on the rest. **Varimax** picks the orthogonal rotation that maximizes the variance of the squared loadings within each column, which pushes every loading towards 0 or towards ±1:

$$
V(\Lambda Q)=\sum_{m=1}^k\Big[\frac1p\sum_{j}\tilde\lambda_{jm}^4-\Big(\frac1p\sum_j\tilde\lambda_{jm}^2\Big)^2\Big]
$$

- $\tilde\lambda_{jm}$: loadings after rotation, $\tilde\Lambda=\Lambda Q$
- $V$: raw varimax criterion. The common Kaiser variant first divides each row by its communality

In the figure's second example, six tests (three maths, three verbal) are fitted with two factors on a correlation matrix from 300 students. Uniquenesses: Algebra 0.361, Geometry 0.423, Physics 0.433, Reading 0.328, Vocab 0.265, Writing 0.473. The maximum residual is 0.017.

The same two-factor fit before and after varimax (rotation of −36.3°). V rises from 0.020 to 0.157; communalities are identical in both.

| Test | Unrotated F1 | Unrotated F2 | Varimax verbal | Varimax maths | h² |
| --- | --- | --- | --- | --- | --- |
| Algebra | 0.59 | 0.54 | 0.15 | **0.79** | 0.64 |
| Geometry | 0.56 | 0.52 | 0.14 | **0.75** | 0.58 |
| Physics | 0.61 | 0.44 | 0.23 | **0.72** | 0.57 |
| Reading | 0.77 | −0.29 | **0.79** | 0.22 | 0.67 |
| Vocab | 0.73 | −0.45 | **0.85** | 0.07 | 0.73 |
| Writing | 0.70 | −0.19 | **0.68** | 0.26 | 0.53 |
| SS loadings | 2.64 | 1.08 | 1.91 | 1.81 | 3.72 |

Before rotation the first factor is a "general" factor that every test loads on, and the second is a bipolar maths-versus-verbal contrast. After varimax the same fit reads as two clean, separate abilities. Neither reading is more *true* than the other, since both reproduce the correlations equally well. Rotation is a choice of coordinates. **Oblique** rotations such as promax and oblimin go further and allow the factors to correlate, which is often more realistic for abilities or attitudes.

> **Do not over-read the signs and order:** Even with one factor, $\lambda$ and $-\lambda$ fit equally well. scikit-learn returned $(-0.894,-0.843,-0.659,-0.895)$ on this data, and the factor score came out as $-0.863$. Factor order and signs are arbitrary, so flip them for presentation and compare solutions only up to rotation.

## Factor analysis vs PCA

Both methods give a low-rank description of a covariance matrix, and both produce "loadings", so they are often confused. The difference is the noise model. PCA, read probabilistically as **probabilistic PCA**, is exactly the FA model with **isotropic** noise $\Psi=\sigma^2I$. FA gives every variable its own $\psi_j$.

$$
\text{FA: }\ \Sigma=\Lambda\Lambda^\top+\operatorname{diag}(\psi_1,\dots,\psi_p)\qquad\text{PPCA: }\ \Sigma=WW^\top+\sigma^2 I
$$

- $\psi_j$: one noise variance per variable, so FA can say one test is noisier than another
- $\sigma^2$: one shared noise variance; the ML solution is the mean of the discarded eigenvalues, and $W$ spans the top eigenvectors

One component vs one factor on the four-test data. PCA loadings are inflated for the noisy Literature test, because PCA has no place to put unique variance except in the component.

| Model | Math | Physics | Literature | Chemistry | Max \|residual\| | Avg log-lik |
| --- | --- | --- | --- | --- | --- | --- |
| **FA** λ | 0.894 | 0.843 | 0.659 | 0.895 | 0.035 | **−4.413** |
| PCA, √(eigenvalue) · u₁ | 0.904 | 0.892 | 0.773 | 0.909 | – | – |
| PPCA W (σ² = 0.321) | 0.855 | 0.843 | 0.731 | 0.860 | 0.144 | −4.528 |

**Key differences**

- **Goal:** PCA keeps the most variance; FA explains the **covariance** (correlations) and treats unique variance as noise.
- **Noise:** PCA/PPCA: one $\sigma^2$ for all variables. FA: a separate $\psi_j$ per variable.
- **Scale:** PCA depends on units: rescale a variable and the components change. FA's ML fit is **scale-equivariant**: fitting the raw covariance here gives loadings $(9.79, 8.09, 7.91, 11.43)$, which are exactly the standardized ones times each column's sd.
- **Solution:** PCA has a closed form (one eigendecomposition) and nested components. FA needs iterative fitting, and its 2-factor solution is not the 1-factor solution plus one more column.
- **Uniqueness:** PCA components are unique up to sign (with distinct eigenvalues). FA factors are unique only up to rotation.

> **When they agree:** If all uniquenesses are roughly equal, or small compared with the communalities, FA and PCA give nearly the same subspace. They diverge when a few variables are much noisier than others. FA down-weights those variables, while PCA lets them pull the components towards themselves.

## In code

```python
import numpy as np

X = np.array([[73,62,83,72],[75,71,79,60],[73,66,70,48],[70,69,67,71],[82,83,77,81],
              [79,75,92,71],[71,52,63,59],[51,52,47,40],[49,58,68,46],[57,58,82,52]], float)
Z = (X - X.mean(0)) / X.std(0)          # standardize -> S is the correlation matrix
S = Z.T @ Z / len(Z)

def factor_em(S, k, iters=5000, tol=1e-6):
  p = S.shape[0]
  L = np.full((p, k), 0.6) if k == 1 else np.linalg.eigh(S)[1][:, ::-1][:, :k]
  psi = np.diag(S) - (L ** 2).sum(1)
  for t in range(iters):
      # E-step: beta = (I + L' Psi^-1 L)^-1 L' Psi^-1
      LtPi = L.T / psi
      beta = np.linalg.solve(np.eye(k) + LtPi @ L, LtPi)
      Ezz = np.eye(k) - beta @ L + beta @ S @ beta.T      # mean of E[zz'|x]
      # M-step
      L_new = S @ beta.T @ np.linalg.inv(Ezz)
      psi = np.diag(S - L_new @ beta @ S)
      done = np.abs(L_new - L).max() < tol
      L = L_new
      if done:
          break
  return L, psi, t + 1

L, psi, n_iter = factor_em(S, k=1)
print(n_iter, L.ravel().round(3), psi.round(3))   # 64 [0.894 0.843 0.659 0.895] [0.201 0.289 0.566 0.199]
print("communalities", (L ** 2).ravel().round(3))
print("residual", np.abs(S - L @ L.T - np.diag(psi)).max().round(3))   # 0.035

# Factor score of a new student
x = (np.array([80, 70, 60, 75]) - X.mean(0)) / X.std(0)
beta = np.linalg.solve(np.eye(1) + (L.T / psi) @ L, L.T / psi)
print("z_hat", (beta @ x).round(3))                  # [0.863]
```

From scratch: EM on the sample covariance, reproducing the worked example (64 iterations, λ = 0.894, 0.843, 0.659, 0.895).

```python
from sklearn.decomposition import FactorAnalysis
from sklearn.preprocessing import StandardScaler

Z = StandardScaler().fit_transform(X)            # population sd, as above
fa = FactorAnalysis(n_components=1, tol=1e-10, max_iter=10000).fit(Z)
print(fa.components_.round(3))      # [[-0.894 -0.843 -0.659 -0.895]]  (sign is arbitrary)
print(fa.noise_variance_.round(3))  # [0.201 0.289 0.566 0.199]
print(fa.score(Z))                  # average log-likelihood -4.413
print(fa.get_covariance().round(3)) # Lambda Lambda' + Psi

# Two factors with varimax on a 6-test battery Y (n x 6):
fa2 = FactorAnalysis(n_components=2, rotation="varimax").fit(Y)
loadings = fa2.components_.T        # rows = tests, columns = factors
scores = fa2.transform(Y)           # posterior means E[z | x]

# Choosing k: compare held-out log-likelihood
# from sklearn.model_selection import cross_val_score
# [cross_val_score(FactorAnalysis(k), Y).mean() for k in range(1, 4)]
```

scikit-learn: same fit (up to sign), plus a varimax-rotated two-factor model.

> **Other libraries:** `factor_analyzer` (Python) and R's `psych::fa` / `factanal` add oblique rotations (promax, oblimin), χ² goodness-of-fit tests, parallel analysis and Heywood-case warnings. These are the tools usually used in psychometrics.

**Strengths**

- You believe a few hidden causes generate the correlations and want to estimate them along with how noisy each variable is.
- Variables have very different reliabilities: FA down-weights noisy ones instead of letting them shape the components.
- You want a proper probabilistic model, with a likelihood for model comparison, posterior factor scores with uncertainty, and missing-data handling via EM.
- Units should not matter: the ML solution is equivariant to rescaling variables.

**Limitations**

- You just want compression or decorrelation: PCA is closed-form, faster, unique and nested.
- The structure is non-linear or the noise is strongly non-Gaussian; consider ICA, autoencoders or non-linear latent models.
- Samples are few relative to variables: the fit is unstable and Heywood cases become common.
- Interpretability is the goal but the rotation is chosen ad hoc; different rotations tell different stories with the same fit.

- **Psychometrics:** The origin of the method (Spearman's *g*, 1904): intelligence and personality batteries, such as the Big Five traits recovered from hundreds of questionnaire items.
- **Surveys and marketing:** Group Likert items into a few constructs (satisfaction, trust, price sensitivity), check that a scale measures one thing, and drop items with high uniqueness.
- **Finance:** Statistical factor models of asset returns: a few market or sector factors plus idiosyncratic risk give a well-conditioned covariance matrix ΛΛᵀ + Ψ for portfolio optimization.
- **Covariance estimation:** With many variables and few samples, a low-rank-plus-diagonal Σ is far more stable than the raw sample covariance. It is used in genomics, sensor arrays and as a Gaussian-model density.

> **Choosing the number of factors:** There is no single right answer. Common tools: the **scree plot** of the correlation eigenvalues (here 3.04, 0.50, 0.27, 0.19, which clearly suggests one factor); **parallel analysis**, which keeps factors whose eigenvalues beat those of random data of the same shape; likelihood-ratio χ² tests; **BIC**; and held-out log-likelihood. The Kaiser rule "eigenvalue > 1" is popular but tends to over-extract. Also check identifiability: the degrees of freedom $\tfrac12[(p-k)^2-(p+k)]$ must be $\ge 0$, so with 4 variables even $k=2$ is already too many ($-1$).

> **Heywood cases:** Sometimes the maximum likelihood wants a uniqueness of $\psi_j\le 0$, which means a communality of 1 or more and a variable perfectly explained by the factors. EM drives $\psi_j$ towards 0 very slowly, and other optimizers clip it at a small bound. This is a symptom, not a result. Typical causes are too few samples, too many factors, or a variable that is nearly a copy of another. Reduce $k$, remove or merge the variable, or use a regularized or Bayesian fit.

> **Factors are not causes:** A good fit shows that the correlations are *consistent* with k common factors. It does not prove that the factors exist or what they mean. Naming a factor is an interpretation, and the rotation you chose shapes it.

**Question:** In a one-factor model on standardized data, two variables have loadings 0.9 and 0.7. What correlation does the model imply between them, and what is the second variable's uniqueness?

- A. 0.63 and 0.51
- B. 0.80 and 0.30
- C. 0.63 and 0.30
- D. 0.16 and 0.51

**Answer:** A. 0.63 and 0.51 — Off-diagonal: $\lambda_1\lambda_2=0.9\times0.7=0.63$. Uniqueness: $1-h^2=1-0.7^2=0.51$.

**Question:** What is the essential modelling difference between factor analysis and probabilistic PCA?

- A. FA uses a non-linear mapping from factors to data
- B. FA allows a different noise variance for each variable; PPCA uses one shared variance
- C. PPCA's latent variables are not Gaussian
- D. FA requires labelled data

**Answer:** B. FA allows a different noise variance for each variable; PPCA uses one shared variance — Both are $x=\mu+\Lambda z+\varepsilon$ with Gaussian $z$. PPCA sets $\Psi=\sigma^2I$ (isotropic), while FA lets $\Psi$ be any diagonal matrix. That is why FA can tell a noisy variable apart from a strongly shared one.

**Question:** You rotate a fitted two-factor solution by 30°. Which of these changes?

- A. The log-likelihood
- B. The communalities
- C. The individual loadings
- D. The uniquenesses

**Answer:** C. The individual loadings — $\Lambda Q(\Lambda Q)^\top=\Lambda\Lambda^\top$, so $\Sigma$, the likelihood, the row norms (communalities) and $\Psi$ are all unchanged. Only how the shared variance is split between the two factors moves.

**Question:** An FA fit reports a uniqueness of 0.001 (communality ≈ 1) for one survey item. The most likely interpretation is…

- A. That item is the best measure of the factor and should be kept as is
- B. A Heywood case: too many factors, too little data, or a near-duplicate item
- C. EM has diverged and the likelihood is decreasing
- D. The data should be standardized

**Answer:** B. A Heywood case: too many factors, too little data, or a near-duplicate item — A uniqueness pinned at (or below) zero is a Heywood case. It signals over-extraction, small samples or redundant variables, not a perfect measurement.

**Key points**

- Model: $x=\mu+\Lambda z+\varepsilon$, $z\sim\mathcal N(0,I)$, $\varepsilon\sim\mathcal N(0,\Psi)$ with $\Psi$ diagonal, so $\Sigma=\Lambda\Lambda^\top+\Psi$.
- Each variance splits into communality $h_j^2=\sum_m\lambda_{jm}^2$ and uniqueness $\psi_j$. Correlations come only from the shared part.
- EM: the E-step computes $\beta=(I+\Lambda^\top\Psi^{-1}\Lambda)^{-1}\Lambda^\top\Psi^{-1}$, $\mathbb E[z|x]=\beta x$, $\operatorname{Cov}(z|x)=I-\beta\Lambda$; the M-step regresses $x$ on the inferred $z$ to update $\Lambda$, then sets $\Psi=\operatorname{diag}(S-\Lambda\beta S)$.
- Loadings are identified only up to rotation; varimax picks the rotation with the simplest structure without changing the fit.
- Versus PCA: diagonal instead of isotropic noise, scale-equivariant, models covariance rather than variance, and down-weights noisy variables.
- Choose k with scree plots, parallel analysis, BIC or held-out likelihood, and treat a uniqueness near 0 (a Heywood case) as a warning.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
