Factor Analysis

Model each feature as a linear combination of a few latent factors plus its own independent noise.

19 min read1563 wordsadvanced0 viewsupdated 2026-10-08

Definition

Factor analysis (FA) is a latent-variable model that explains the correlations among pp observed variables with a small number kk of unobserved factors. Each variable is a weighted sum of the factors plus a noise term of its own, and the noise terms are independent of each other. All the covariance between variables is therefore carried by the factors.

At a glance

Task
Unsupervised: latent structure, dimensionality reduction, covariance modelling
Input → output
X∈Rn×pX\in\mathbb{R}^{n\times p} → loadings Λ∈Rp×k\Lambda\in\mathbb{R}^{p\times k}, uniquenesses ψ1,…,ψp\psi_1,\dots,\psi_p, and factor scores z^∈Rk\hat z\in\mathbb{R}^{k} per row
Learns
mean μ\mu, loading matrix Λ\Lambda, diagonal noise covariance Ψ\Psi, by maximum likelihood
Training
EM (or direct likelihood optimization); each iteration costs O(p2k)O(p^2k) once the p×pp\times p covariance has been formed in O(np2)O(np^2)
Prediction
factor scores z^=β(x−μ)\hat z=\beta(x-\mu), which costs O(pk)O(pk)
Hyperparameters
number of factors kk, rotation (none, varimax, quartimax, oblique), whether to work on correlations or covariances
Assumes
linear effects of the factors, Gaussian factors and noise, noise independent across variables

The problem

A class sits four tests: Math, Physics, Literature and Chemistry. The scores are all positively correlated. Students who do well in Math tend to do well in Chemistry (r=0.81r=0.81), and even Literature correlates 0.56–0.59 with the others. One natural explanation is that a single hidden trait, call it general academic ability, pushes all four scores up or down together, and that each test then adds its own luck and test-specific skill on top.

We cannot measure that trait directly. Factor analysis asks whether such a trait is consistent with the data, how strongly each test reflects it, and how much of each test is just its own noise. The same question comes up whenever many measurements look like noisy readings of a few underlying quantities: survey items and attitudes, stock returns and market sectors, gene expression and pathways.

The idea

Imagine a generative story. First draw each student's factor values zz, such as ability. Then produce each test score as a weighted sum of those factors plus independent noise. Because every test shares the same zz, the tests end up correlated. Because the noise is private to each test, it adds variance to that test but no covariance with any other test.

Fitting the model runs that story backwards. We look for the weights (loadings) and noise levels that make the model's implied covariance matrix as close as possible to the observed one. The split is the key output: each variable's variance is divided into a shared part, which the factors explain, and a unique part, which they do not.

Vocabulary

Factor (latent variable)
an unobserved quantity zmz_m; by convention z∼N(0,I)z\sim\mathcal N(0,I), so factors have unit variance and are uncorrelated
Loading
λjm\lambda_{jm}, the weight of factor mm in variable jj; on standardized data it is the correlation between the variable and the factor
Communality
hj2=∑mλjm2h_j^2=\sum_m\lambda_{jm}^2, the part of variable jj's variance that the factors explain
Uniqueness
ψj\psi_j, the variance of variable jj's own noise (specific factor plus measurement error)
Factor score
an estimate of a row's zz, usually the posterior mean E[z∣x]\mathbb E[z\mid x]
Rotation
replacing Λ\Lambda by ΛQ\Lambda Q for an orthogonal QQ; the fit does not change, but the loadings become easier to interpret

The model

x=μ+Λz+ε,z∼N(0,Ik),ε∼N(0,Ψ),Ψ=diag⁡(ψ1,…,ψp)x=\mu+\Lambda z+\varepsilon,\qquad z\sim\mathcal N(0,I_k),\qquad \varepsilon\sim\mathcal N(0,\Psi),\quad \Psi=\operatorname{diag}(\psi_1,\dots,\psi_p)
x∈Rpx\in\mathbb R^p
one observed row, for example one student's four test scores
μ\mu
mean of each variable
Λ∈Rp×k\Lambda\in\mathbb R^{p\times k}
factor loading matrix, one row per variable and one column per factor
z∈Rkz\in\mathbb R^k
latent factors, independent of ε\varepsilon, with k≪pk\ll p
ε\varepsilon
unique noise; Ψ\Psi is diagonal, so the noise terms are independent across variables

Since xx is a linear function of Gaussians, it is Gaussian too. Its covariance is the sum of a low-rank part that comes from the factors and a diagonal part that comes from the noise:

Σ=Cov⁡(x)=ΛΛ⊤+Ψ,x∼N(μ,  ΛΛ⊤+Ψ)\Sigma=\operatorname{Cov}(x)=\Lambda\Lambda^\top+\Psi,\qquad x\sim\mathcal N(\mu,\;\Lambda\Lambda^\top+\Psi)
ΛΛ⊤\Lambda\Lambda^\top
rank-kk shared covariance; its off-diagonal entries are the only source of correlation between variables
Ψ\Psi
diagonal; it adds to the variances and never to the covariances
Σjj=∑mλjm2⏟hj2 (communality)+ψj⏟uniqueness,Σij=∑mλimλjm(i≠j)\Sigma_{jj}=\underbrace{\textstyle\sum_m\lambda_{jm}^2}_{h_j^2\ \text{(communality)}}+\underbrace{\psi_j}_{\text{uniqueness}},\qquad \Sigma_{ij}=\textstyle\sum_m\lambda_{im}\lambda_{jm}\quad(i\neq j)
hj2h_j^2
variance of variable j explained by the common factors
ψj\psi_j
variance left over; on standardized data hj2+ψj=1h_j^2+\psi_j=1

With one factor the off-diagonal structure is very rigid: Corr⁡(xi,xj)=λiλj\operatorname{Corr}(x_i,x_j)=\lambda_i\lambda_j. This is a testable claim. A covariance matrix that fits it is one where every correlation is the product of two per-variable numbers. Counting parameters makes the same point: Σ\Sigma has p(p+1)/2p(p+1)/2 free entries, and the model uses pk+p−k(k−1)/2pk+p-k(k-1)/2 of them, which leaves 12[(p−k)2−(p+k)]\tfrac12[(p-k)^2-(p+k)] degrees of freedom. For 4 tests and 1 factor that is 2, so the model genuinely constrains the data.

Fitting by EM

The log-likelihood of centred data with sample covariance SS is ℓ=−n2[plog⁡2π+log⁡∣Σ∣+tr⁡(Σ−1S)]\ell=-\tfrac n2\big[p\log 2\pi+\log|\Sigma|+\operatorname{tr}(\Sigma^{-1}S)\big] with Σ=ΛΛ⊤+Ψ\Sigma=\Lambda\Lambda^\top+\Psi. There is no closed form for its maximum. If we knew each row's zz, though, the problem would reduce to pp ordinary linear regressions of xjx_j on zz. Expectation–maximization exploits this: it alternates between inferring zz given the current parameters and refitting the regressions using those inferred values.

E-step. Under the current Λ,Ψ\Lambda,\Psi the pair (z,x)(z,x) is jointly Gaussian, so the posterior of zz given xx is Gaussian with:

β=Λ⊤(ΛΛ⊤+Ψ)−1=(I+Λ⊤Ψ−1Λ)−1Λ⊤Ψ−1\beta=\Lambda^\top(\Lambda\Lambda^\top+\Psi)^{-1}=(I+\Lambda^\top\Psi^{-1}\Lambda)^{-1}\Lambda^\top\Psi^{-1}
β∈Rk×p\beta\in\mathbb R^{k\times p}
regression of zz on xx. The second form uses the Woodbury identity and only inverts a k×kk\times k matrix plus the diagonal Ψ\Psi
E[z∣x]=β(x−μ),E[zz⊤∣x]=I−βΛ⏟Cov⁡(z∣x)+β(x−μ)(x−μ)⊤β⊤\mathbb E[z\mid x]=\beta(x-\mu),\qquad \mathbb E[zz^\top\mid x]=\underbrace{I-\beta\Lambda}_{\operatorname{Cov}(z\mid x)}+\beta(x-\mu)(x-\mu)^\top\beta^\top
E[z∣x]\mathbb E[z\mid x]
posterior mean of the factors. This is also the regression (Thomson) factor score
I−βΛI-\beta\Lambda
posterior covariance. It is the same for every row and shrinks as the loadings grow

M-step. Maximize the expected complete-data log-likelihood. This is the least-squares regression of xx on zz, with zz⊤zz^\top replaced by its expectation. Averaged over rows it depends on the data only through SS:

Λnew=Sβ⊤(I−βΛ+βSβ⊤)−1,Ψnew=diag⁡ ⁣(S−Λnew βS)\Lambda_{\text{new}}=S\beta^\top\big(I-\beta\Lambda+\beta S\beta^\top\big)^{-1},\qquad \Psi_{\text{new}}=\operatorname{diag}\!\big(S-\Lambda_{\text{new}}\,\beta S\big)
Sβ⊤S\beta^\top
1n∑i(xi−μ) E[z∣xi]⊤\tfrac1n\sum_i (x_i-\mu)\,\mathbb E[z\mid x_i]^\top, the cross-covariance of the data and the inferred factors
I−βΛ+βSβ⊤I-\beta\Lambda+\beta S\beta^\top
1n∑iE[zz⊤∣xi]\tfrac1n\sum_i\mathbb E[zz^\top\mid x_i], the expected second moment of the factors
diag⁡(⋅)\operatorname{diag}(\cdot)
keep only the diagonal. This is where the assumption that Ψ\Psi is diagonal enters

Each iteration is guaranteed not to decrease the likelihood. Convergence is usually linear and can be slow when a uniqueness heads towards 0. Production code such as scikit-learn's FactorAnalysis uses an equivalent SVD-based update, and statistics packages often maximize ℓ\ell directly with a quasi-Newton method. The fitted model is the same either way.

EM for factor analysis

  1. 1Centre the data (standardize it too if you want to work with correlations) and form SS.
  2. 2Initialize Λ\Lambda (for example from the top-kk PCA loadings, or a constant) and Ψ=diag⁡(S)−diag⁡(ΛΛ⊤)\Psi=\operatorname{diag}(S)-\operatorname{diag}(\Lambda\Lambda^\top), keeping every ψj>0\psi_j>0.
  3. 3E-step: β=(I+Λ⊤Ψ−1Λ)−1Λ⊤Ψ−1\beta=(I+\Lambda^\top\Psi^{-1}\Lambda)^{-1}\Lambda^\top\Psi^{-1}.
  4. 4M-step: Λ←Sβ⊤(I−βΛ+βSβ⊤)−1\Lambda\leftarrow S\beta^\top(I-\beta\Lambda+\beta S\beta^\top)^{-1}, then Ψ←diag⁡(S−ΛβS)\Psi\leftarrow\operatorname{diag}(S-\Lambda\beta S).
  5. 5Repeat until the loadings (or ℓ\ell) stop changing. Then rotate if you want interpretable factors, and score rows with z^=β(x−μ)\hat z=\beta(x-\mu).

Worked example: one factor behind four tests

Ten students and four tests. We standardize each column (population standard deviation), so SS is the correlation matrix RR and the loadings are correlations with the factor.

StudentMathPhysicsLiteratureChemistry
173628372
275717960
373667048
470696771
582837781
679759271
771526359
851524740
949586846
1057588252
Raw scores. Column means 68.0, 64.6, 72.8, 60.0; standard deviations 10.95, 9.59, 12.00, 12.78.
rMathPhysicsLiteratureChemistry
Math10.7560.5620.806
Physics0.75610.5910.743
Literature0.5620.59110.594
Chemistry0.8060.7430.5941
Sample correlation matrix R. Literature is the least correlated with the rest.

The first EM iteration by hand

  1. 1Start with every λj=0.6\lambda_j=0.6 and ψj=0.64\psi_j=0.64, so each variance is 0.36+0.64=10.36+0.64=1. The model correlation is 0.62=0.360.6^2=0.36 everywhere, far below the observed values. Average log-likelihood: −4.861-4.861.
  2. 2E-step. wj=λj/ψj=0.9375w_j=\lambda_j/\psi_j=0.9375, and 1+∑jλjwj=1+4(0.6)(0.9375)=3.251+\sum_j\lambda_jw_j=1+4(0.6)(0.9375)=3.25, so βj=0.9375/3.25=0.2885\beta_j=0.9375/3.25=0.2885 for every test. The posterior variance of a student's factor is 1−βΛ=1−4(0.2885)(0.6)=0.3081-\beta\Lambda=1-4(0.2885)(0.6)=0.308.
  3. 3M-step, numerator. (Sβ⊤)j=0.2885×(row sum of R)(S\beta^\top)_j=0.2885\times(\text{row sum of }R). For Math that is 0.2885×3.123=0.9010.2885\times3.123=0.901.
  4. 4M-step, denominator. 1−βΛ+βRβ⊤=0.308+0.28852×(sum of all entries of R=12.10)=0.308+1.007=1.3151-\beta\Lambda+\beta R\beta^\top=0.308+0.2885^2\times(\text{sum of all entries of }R=12.10)=0.308+1.007=1.315.
  5. 5New loadings λ=(0.685,  0.678,  0.603,  0.690)\lambda=(0.685,\;0.678,\;0.603,\;0.690); new uniquenesses ψ=(0.383,  0.396,  0.523,  0.375)\psi=(0.383,\;0.396,\;0.523,\;0.375). Log-likelihood rises to −4.565-4.565.
  6. 6Iterate. After 20 rounds every loading is within 0.007 of its final value; after 64 rounds the largest change is below 10−610^{-6} and ℓ=−4.413\ell=-4.413.
TestLoading λCommunality h² = λ²Uniqueness ψScore weight w = λ/ψ
Math0.8940.7990.2014.46
Physics0.8430.7110.2892.91
Literature0.6590.4340.5661.17
Chemistry0.8950.8010.1994.49
Converged one-factor solution. Communality + uniqueness = 1 for each test. The factor explains 68.6% of the total variance (2.745 of 4).

Literature is the outlier: more than half its variance (0.566) is unique, meaning it is driven by something the other three tests do not share. Math and Chemistry are the cleanest measures of the common factor.

PairObserved rModel λᵢλⱼResidual
Math · Physics0.7560.754+0.002
Math · Literature0.5620.589−0.028
Math · Chemistry0.8060.800+0.006
Physics · Literature0.5910.556+0.035
Physics · Chemistry0.7430.754−0.011
Literature · Chemistry0.5940.590+0.005
Observed vs reconstructed correlations, Σ^ij=λiλj\hat\Sigma_{ij}=\lambda_i\lambda_j. The diagonal is reproduced exactly (h2+ψ=1h^2+\psi=1); every off-diagonal residual is within ±0.035.

Scoring a new student. Take scores (80, 70, 60, 75). As z-scores that is (1.095,  0.563,  −1.067,  1.174)(1.095,\;0.563,\;-1.067,\;1.174). The factor score is z^=∑jwjxj / (1+∑jλjwj)=10.55/12.23=\hat z=\sum_j w_jx_j\,/\,(1+\sum_j\lambda_jw_j)=10.55/12.23= 0.863, with posterior standard deviation 1/12.23=0.286\sqrt{1/12.23}=0.286. The weak Literature mark contributes only −1.24-1.24 against +4.88+4.88 (Math) and +5.27+5.27 (Chemistry), because Literature is mostly noise as far as the factor is concerned. FA weights each variable by its reliability, λj/ψj\lambda_j/\psi_j.

▸ factor analysis · EMstep 1/10
each bar = variance 1: ■ communality λ² ■ uniqueness ψtestλψMath0.6000.640Physics0.6000.640Literature0.6000.640Chemistry0.6000.6401residual R − ΛΛᵀ − ΨMPLC·.40.20.45.40·.23.38.20.23·.23.45.38.23·average log-likelihood per student vs iteration-4.41-4.86064t=0 · -4.861
iteration 0log-lik -4.861explained 36.0%max |resid| 0.446
  1. 1λ ← 0.6, ψ ← 0.64 # standardized tests
  2. 2repeat:
  3. 3 E: w = λ/ψ, β = w / (1 + λ·w)
  4. 4 E[z|x] = βx, Var[z|x] = 1 − βλ
  5. 5 M: λ ← Rβᵀ / (1 − βλ + βRβᵀ)
  6. 6 ψ ← diag(R − λβR)
  7. 7until max|Δλ| < 1e-6
  8. 8score: ẑ = βx_new

Start every loading at 0.6 and every uniqueness at 0.64, so each bar sums to 1. The model correlation is 0.36 everywhere, far below the observed 0.56–0.81 (red residuals). Log-likelihood -4.861.

fig.Tab 1 runs EM on the four-test example. The bars split each test's unit variance into communality λ² and uniqueness ψ, the heatmap shows the residual R − ΛΛᵀ − Ψ shrinking from about 0.4 to within ±0.035, and the log-likelihood climbs from −4.861 to −4.413 in 64 iterations. The last frame scores the new student (ẑ = 0.863). Tab 2 fits two factors to six tests. Drag the F1 axis to rotate the solution: the loadings change, but the communalities and the fit do not. Varimax at −36.3° gives a clean verbal factor and a clean maths factor.

Rotational indeterminacy and varimax

With k≥2k\ge2 the loadings are not unique. For any orthogonal k×kk\times k matrix QQ, setting Λ~=ΛQ\tilde\Lambda=\Lambda Q and z~=Q⊤z\tilde z=Q^\top z gives exactly the same distribution for xx:

Λ~Λ~⊤=ΛQQ⊤Λ⊤=ΛΛ⊤  ⇒  Σ~=Σ\tilde\Lambda\tilde\Lambda^\top=\Lambda QQ^\top\Lambda^\top=\Lambda\Lambda^\top\;\Rightarrow\;\tilde\Sigma=\Sigma
QQ
any rotation or reflection, QQ⊤=IQQ^\top=I
hj2h_j^2
row norms of Λ\Lambda; these are unchanged by QQ, so communalities and uniquenesses are identified while individual loadings are not

The likelihood cannot choose among these solutions, so we choose one that is easy to interpret. The aim is simple structure: each variable loads strongly on one factor and weakly on the rest. Varimax picks the orthogonal rotation that maximizes the variance of the squared loadings within each column, which pushes every loading towards 0 or towards ±1:

V(ΛQ)=∑m=1k[1p∑jλ~jm4−(1p∑jλ~jm2)2]V(\Lambda Q)=\sum_{m=1}^k\Big[\frac1p\sum_{j}\tilde\lambda_{jm}^4-\Big(\frac1p\sum_j\tilde\lambda_{jm}^2\Big)^2\Big]
λ~jm\tilde\lambda_{jm}
loadings after rotation, Λ~=ΛQ\tilde\Lambda=\Lambda Q
VV
raw varimax criterion. The common Kaiser variant first divides each row by its communality

In the figure's second example, six tests (three maths, three verbal) are fitted with two factors on a correlation matrix from 300 students. Uniquenesses: Algebra 0.361, Geometry 0.423, Physics 0.433, Reading 0.328, Vocab 0.265, Writing 0.473. The maximum residual is 0.017.

TestUnrotated F1Unrotated F2Varimax verbalVarimax mathsh²
Algebra0.590.540.150.790.64
Geometry0.560.520.140.750.58
Physics0.610.440.230.720.57
Reading0.77−0.290.790.220.67
Vocab0.73−0.450.850.070.73
Writing0.70−0.190.680.260.53
SS loadings2.641.081.911.813.72
The same two-factor fit before and after varimax (rotation of −36.3°). V rises from 0.020 to 0.157; communalities are identical in both.

Before rotation the first factor is a "general" factor that every test loads on, and the second is a bipolar maths-versus-verbal contrast. After varimax the same fit reads as two clean, separate abilities. Neither reading is more true than the other, since both reproduce the correlations equally well. Rotation is a choice of coordinates. Oblique rotations such as promax and oblimin go further and allow the factors to correlate, which is often more realistic for abilities or attitudes.

Factor analysis vs PCA

Both methods give a low-rank description of a covariance matrix, and both produce "loadings", so they are often confused. The difference is the noise model. PCA, read probabilistically as probabilistic PCA, is exactly the FA model with isotropic noise Ψ=σ2I\Psi=\sigma^2I. FA gives every variable its own ψj\psi_j.

FA:  Σ=ΛΛ⊤+diag⁡(ψ1,…,ψp)PPCA:  Σ=WW⊤+σ2I\text{FA: }\ \Sigma=\Lambda\Lambda^\top+\operatorname{diag}(\psi_1,\dots,\psi_p)\qquad\text{PPCA: }\ \Sigma=WW^\top+\sigma^2 I
ψj\psi_j
one noise variance per variable, so FA can say one test is noisier than another
σ2\sigma^2
one shared noise variance; the ML solution is the mean of the discarded eigenvalues, and WW spans the top eigenvectors
ModelMathPhysicsLiteratureChemistryMax |residual|Avg log-lik
FA λ0.8940.8430.6590.8950.035−4.413
PCA, √(eigenvalue) · u₁0.9040.8920.7730.909––
PPCA W (σ² = 0.321)0.8550.8430.7310.8600.144−4.528
One component vs one factor on the four-test data. PCA loadings are inflated for the noisy Literature test, because PCA has no place to put unique variance except in the component.

Key differences

Goal
PCA keeps the most variance; FA explains the covariance (correlations) and treats unique variance as noise.
Noise
PCA/PPCA: one σ2\sigma^2 for all variables. FA: a separate ψj\psi_j per variable.
Scale
PCA depends on units: rescale a variable and the components change. FA's ML fit is scale-equivariant: fitting the raw covariance here gives loadings (9.79,8.09,7.91,11.43)(9.79, 8.09, 7.91, 11.43), which are exactly the standardized ones times each column's sd.
Solution
PCA has a closed form (one eigendecomposition) and nested components. FA needs iterative fitting, and its 2-factor solution is not the 1-factor solution plus one more column.
Uniqueness
PCA components are unique up to sign (with distinct eigenvalues). FA factors are unique only up to rotation.

In code

python
1import numpy as np
2
3X = np.array([[73,62,83,72],[75,71,79,60],[73,66,70,48],[70,69,67,71],[82,83,77,81],
4 [79,75,92,71],[71,52,63,59],[51,52,47,40],[49,58,68,46],[57,58,82,52]], float)
5Z = (X - X.mean(0)) / X.std(0) # standardize -> S is the correlation matrix
6S = Z.T @ Z / len(Z)
7
8def factor_em(S, k, iters=5000, tol=1e-6):
9 p = S.shape[0]
10 L = np.full((p, k), 0.6) if k == 1 else np.linalg.eigh(S)[1][:, ::-1][:, :k]
11 psi = np.diag(S) - (L ** 2).sum(1)
12 for t in range(iters):
13 # E-step: beta = (I + L' Psi^-1 L)^-1 L' Psi^-1
14 LtPi = L.T / psi
15 beta = np.linalg.solve(np.eye(k) + LtPi @ L, LtPi)
16 Ezz = np.eye(k) - beta @ L + beta @ S @ beta.T # mean of E[zz'|x]
17 # M-step
18 L_new = S @ beta.T @ np.linalg.inv(Ezz)
19 psi = np.diag(S - L_new @ beta @ S)
20 done = np.abs(L_new - L).max() < tol
21 L = L_new
22 if done:
23 break
24 return L, psi, t + 1
25
26L, psi, n_iter = factor_em(S, k=1)
27print(n_iter, L.ravel().round(3), psi.round(3)) # 64 [0.894 0.843 0.659 0.895] [0.201 0.289 0.566 0.199]
28print("communalities", (L ** 2).ravel().round(3))
29print("residual", np.abs(S - L @ L.T - np.diag(psi)).max().round(3)) # 0.035
30
31# Factor score of a new student
32x = (np.array([80, 70, 60, 75]) - X.mean(0)) / X.std(0)
33beta = np.linalg.solve(np.eye(1) + (L.T / psi) @ L, L.T / psi)
34print("z_hat", (beta @ x).round(3)) # [0.863]
From scratch: EM on the sample covariance, reproducing the worked example (64 iterations, λ = 0.894, 0.843, 0.659, 0.895).
python
1from sklearn.decomposition import FactorAnalysis
2from sklearn.preprocessing import StandardScaler
3
4Z = StandardScaler().fit_transform(X) # population sd, as above
5fa = FactorAnalysis(n_components=1, tol=1e-10, max_iter=10000).fit(Z)
6print(fa.components_.round(3)) # [[-0.894 -0.843 -0.659 -0.895]] (sign is arbitrary)
7print(fa.noise_variance_.round(3)) # [0.201 0.289 0.566 0.199]
8print(fa.score(Z)) # average log-likelihood -4.413
9print(fa.get_covariance().round(3)) # Lambda Lambda' + Psi
10
11# Two factors with varimax on a 6-test battery Y (n x 6):
12fa2 = FactorAnalysis(n_components=2, rotation="varimax").fit(Y)
13loadings = fa2.components_.T # rows = tests, columns = factors
14scores = fa2.transform(Y) # posterior means E[z | x]
15
16# Choosing k: compare held-out log-likelihood
17# from sklearn.model_selection import cross_val_score
18# [cross_val_score(FactorAnalysis(k), Y).mean() for k in range(1, 4)]
scikit-learn: same fit (up to sign), plus a varimax-rotated two-factor model.

Use when

  • +You believe a few hidden causes generate the correlations and want to estimate them along with how noisy each variable is.
  • +Variables have very different reliabilities: FA down-weights noisy ones instead of letting them shape the components.
  • +You want a proper probabilistic model, with a likelihood for model comparison, posterior factor scores with uncertainty, and missing-data handling via EM.
  • +Units should not matter: the ML solution is equivariant to rescaling variables.

Avoid when

  • −You just want compression or decorrelation: PCA is closed-form, faster, unique and nested.
  • −The structure is non-linear or the noise is strongly non-Gaussian; consider ICA, autoencoders or non-linear latent models.
  • −Samples are few relative to variables: the fit is unstable and Heywood cases become common.
  • −Interpretability is the goal but the rotation is chosen ad hoc; different rotations tell different stories with the same fit.

Where it's used

Psychometrics

The origin of the method (Spearman's g, 1904): intelligence and personality batteries, such as the Big Five traits recovered from hundreds of questionnaire items.

Surveys and marketing

Group Likert items into a few constructs (satisfaction, trust, price sensitivity), check that a scale measures one thing, and drop items with high uniqueness.

Finance

Statistical factor models of asset returns: a few market or sector factors plus idiosyncratic risk give a well-conditioned covariance matrix ΛΛᵀ + Ψ for portfolio optimization.

Covariance estimation

With many variables and few samples, a low-rank-plus-diagonal Σ is far more stable than the raw sample covariance. It is used in genomics, sensor arrays and as a Gaussian-model density.

Check yourself

In a one-factor model on standardized data, two variables have loadings 0.9 and 0.7. What correlation does the model imply between them, and what is the second variable's uniqueness?

Check yourself

What is the essential modelling difference between factor analysis and probabilistic PCA?

Check yourself

You rotate a fitted two-factor solution by 30°. Which of these changes?

Check yourself

An FA fit reports a uniqueness of 0.001 (communality ≈ 1) for one survey item. The most likely interpretation is…

Remember

  • ✓Model: x=μ+Λz+εx=\mu+\Lambda z+\varepsilon, z∼N(0,I)z\sim\mathcal N(0,I), ε∼N(0,Ψ)\varepsilon\sim\mathcal N(0,\Psi) with Ψ\Psi diagonal, so Σ=ΛΛ⊤+Ψ\Sigma=\Lambda\Lambda^\top+\Psi.
  • ✓Each variance splits into communality hj2=∑mλjm2h_j^2=\sum_m\lambda_{jm}^2 and uniqueness ψj\psi_j. Correlations come only from the shared part.
  • ✓EM: the E-step computes β=(I+Λ⊤Ψ−1Λ)−1Λ⊤Ψ−1\beta=(I+\Lambda^\top\Psi^{-1}\Lambda)^{-1}\Lambda^\top\Psi^{-1}, E[z∣x]=βx\mathbb E[z|x]=\beta x, Cov⁡(z∣x)=I−βΛ\operatorname{Cov}(z|x)=I-\beta\Lambda; the M-step regresses xx on the inferred zz to update Λ\Lambda, then sets Ψ=diag⁡(S−ΛβS)\Psi=\operatorname{diag}(S-\Lambda\beta S).
  • ✓Loadings are identified only up to rotation; varimax picks the rotation with the simplest structure without changing the fit.
  • ✓Versus PCA: diagonal instead of isotropic noise, scale-equivariant, models covariance rather than variance, and down-weights noisy variables.
  • ✓Choose k with scree plots, parallel analysis, BIC or held-out likelihood, and treat a uniqueness near 0 (a Heywood case) as a warning.
← back to ML Algorithms roadmap