Definition
Factor analysis (FA) is a latent-variable model that explains the correlations among observed variables with a small number of unobserved factors. Each variable is a weighted sum of the factors plus a noise term of its own, and the noise terms are independent of each other. All the covariance between variables is therefore carried by the factors.
At a glance
- Task
- Unsupervised: latent structure, dimensionality reduction, covariance modelling
- Input → output
- → loadings , uniquenesses , and factor scores per row
- Learns
- mean , loading matrix , diagonal noise covariance , by maximum likelihood
- Training
- EM (or direct likelihood optimization); each iteration costs once the covariance has been formed in
- Prediction
- factor scores , which costs
- Hyperparameters
- number of factors , rotation (none, varimax, quartimax, oblique), whether to work on correlations or covariances
- Assumes
- linear effects of the factors, Gaussian factors and noise, noise independent across variables
The problem
A class sits four tests: Math, Physics, Literature and Chemistry. The scores are all positively correlated. Students who do well in Math tend to do well in Chemistry (), and even Literature correlates 0.56–0.59 with the others. One natural explanation is that a single hidden trait, call it general academic ability, pushes all four scores up or down together, and that each test then adds its own luck and test-specific skill on top.
We cannot measure that trait directly. Factor analysis asks whether such a trait is consistent with the data, how strongly each test reflects it, and how much of each test is just its own noise. The same question comes up whenever many measurements look like noisy readings of a few underlying quantities: survey items and attitudes, stock returns and market sectors, gene expression and pathways.
The idea
Imagine a generative story. First draw each student's factor values , such as ability. Then produce each test score as a weighted sum of those factors plus independent noise. Because every test shares the same , the tests end up correlated. Because the noise is private to each test, it adds variance to that test but no covariance with any other test.
Fitting the model runs that story backwards. We look for the weights (loadings) and noise levels that make the model's implied covariance matrix as close as possible to the observed one. The split is the key output: each variable's variance is divided into a shared part, which the factors explain, and a unique part, which they do not.
Vocabulary
- Factor (latent variable)
- an unobserved quantity ; by convention , so factors have unit variance and are uncorrelated
- Loading
- , the weight of factor in variable ; on standardized data it is the correlation between the variable and the factor
- Communality
- , the part of variable 's variance that the factors explain
- Uniqueness
- , the variance of variable 's own noise (specific factor plus measurement error)
- Factor score
- an estimate of a row's , usually the posterior mean
- Rotation
- replacing by for an orthogonal ; the fit does not change, but the loadings become easier to interpret
The model
- one observed row, for example one student's four test scores
- mean of each variable
- factor loading matrix, one row per variable and one column per factor
- latent factors, independent of , with
- unique noise; is diagonal, so the noise terms are independent across variables
Since is a linear function of Gaussians, it is Gaussian too. Its covariance is the sum of a low-rank part that comes from the factors and a diagonal part that comes from the noise:
- rank- shared covariance; its off-diagonal entries are the only source of correlation between variables
- diagonal; it adds to the variances and never to the covariances
- variance of variable j explained by the common factors
- variance left over; on standardized data
With one factor the off-diagonal structure is very rigid: . This is a testable claim. A covariance matrix that fits it is one where every correlation is the product of two per-variable numbers. Counting parameters makes the same point: has free entries, and the model uses of them, which leaves degrees of freedom. For 4 tests and 1 factor that is 2, so the model genuinely constrains the data.
Fitting by EM
The log-likelihood of centred data with sample covariance is with . There is no closed form for its maximum. If we knew each row's , though, the problem would reduce to ordinary linear regressions of on . Expectation–maximization exploits this: it alternates between inferring given the current parameters and refitting the regressions using those inferred values.
E-step. Under the current the pair is jointly Gaussian, so the posterior of given is Gaussian with:
- regression of on . The second form uses the Woodbury identity and only inverts a matrix plus the diagonal
- posterior mean of the factors. This is also the regression (Thomson) factor score
- posterior covariance. It is the same for every row and shrinks as the loadings grow
M-step. Maximize the expected complete-data log-likelihood. This is the least-squares regression of on , with replaced by its expectation. Averaged over rows it depends on the data only through :
- , the cross-covariance of the data and the inferred factors
- , the expected second moment of the factors
- keep only the diagonal. This is where the assumption that is diagonal enters
Each iteration is guaranteed not to decrease the likelihood. Convergence is usually linear and can be slow when a uniqueness heads towards 0. Production code such as scikit-learn's FactorAnalysis uses an equivalent SVD-based update, and statistics packages often maximize directly with a quasi-Newton method. The fitted model is the same either way.
EM for factor analysis
- 1Centre the data (standardize it too if you want to work with correlations) and form .
- 2Initialize (for example from the top- PCA loadings, or a constant) and , keeping every .
- 3E-step: .
- 4M-step: , then .
- 5Repeat until the loadings (or ) stop changing. Then rotate if you want interpretable factors, and score rows with .
Worked example: one factor behind four tests
Ten students and four tests. We standardize each column (population standard deviation), so is the correlation matrix and the loadings are correlations with the factor.
| Student | Math | Physics | Literature | Chemistry |
|---|---|---|---|---|
| 1 | 73 | 62 | 83 | 72 |
| 2 | 75 | 71 | 79 | 60 |
| 3 | 73 | 66 | 70 | 48 |
| 4 | 70 | 69 | 67 | 71 |
| 5 | 82 | 83 | 77 | 81 |
| 6 | 79 | 75 | 92 | 71 |
| 7 | 71 | 52 | 63 | 59 |
| 8 | 51 | 52 | 47 | 40 |
| 9 | 49 | 58 | 68 | 46 |
| 10 | 57 | 58 | 82 | 52 |
| r | Math | Physics | Literature | Chemistry |
|---|---|---|---|---|
| Math | 1 | 0.756 | 0.562 | 0.806 |
| Physics | 0.756 | 1 | 0.591 | 0.743 |
| Literature | 0.562 | 0.591 | 1 | 0.594 |
| Chemistry | 0.806 | 0.743 | 0.594 | 1 |
The first EM iteration by hand
- 1Start with every and , so each variance is . The model correlation is everywhere, far below the observed values. Average log-likelihood: .
- 2E-step. , and , so for every test. The posterior variance of a student's factor is .
- 3M-step, numerator. . For Math that is .
- 4M-step, denominator. .
- 5New loadings ; new uniquenesses . Log-likelihood rises to .
- 6Iterate. After 20 rounds every loading is within 0.007 of its final value; after 64 rounds the largest change is below and .
| Test | Loading λ | Communality h² = λ² | Uniqueness ψ | Score weight w = λ/ψ |
|---|---|---|---|---|
| Math | 0.894 | 0.799 | 0.201 | 4.46 |
| Physics | 0.843 | 0.711 | 0.289 | 2.91 |
| Literature | 0.659 | 0.434 | 0.566 | 1.17 |
| Chemistry | 0.895 | 0.801 | 0.199 | 4.49 |
Literature is the outlier: more than half its variance (0.566) is unique, meaning it is driven by something the other three tests do not share. Math and Chemistry are the cleanest measures of the common factor.
| Pair | Observed r | Model λᵢλⱼ | Residual |
|---|---|---|---|
| Math · Physics | 0.756 | 0.754 | +0.002 |
| Math · Literature | 0.562 | 0.589 | −0.028 |
| Math · Chemistry | 0.806 | 0.800 | +0.006 |
| Physics · Literature | 0.591 | 0.556 | +0.035 |
| Physics · Chemistry | 0.743 | 0.754 | −0.011 |
| Literature · Chemistry | 0.594 | 0.590 | +0.005 |
Scoring a new student. Take scores (80, 70, 60, 75). As z-scores that is . The factor score is 0.863, with posterior standard deviation . The weak Literature mark contributes only against (Math) and (Chemistry), because Literature is mostly noise as far as the factor is concerned. FA weights each variable by its reliability, .
- 1λ ← 0.6, ψ ← 0.64 # standardized tests
- 2repeat:
- 3 E: w = λ/ψ, β = w / (1 + λ·w)
- 4 E[z|x] = βx, Var[z|x] = 1 − βλ
- 5 M: λ ← Rβᵀ / (1 − βλ + βRβᵀ)
- 6 ψ ← diag(R − λβR)
- 7until max|Δλ| < 1e-6
- 8score: ẑ = βx_new
Start every loading at 0.6 and every uniqueness at 0.64, so each bar sums to 1. The model correlation is 0.36 everywhere, far below the observed 0.56–0.81 (red residuals). Log-likelihood -4.861.
Rotational indeterminacy and varimax
With the loadings are not unique. For any orthogonal matrix , setting and gives exactly the same distribution for :
- any rotation or reflection,
- row norms of ; these are unchanged by , so communalities and uniquenesses are identified while individual loadings are not
The likelihood cannot choose among these solutions, so we choose one that is easy to interpret. The aim is simple structure: each variable loads strongly on one factor and weakly on the rest. Varimax picks the orthogonal rotation that maximizes the variance of the squared loadings within each column, which pushes every loading towards 0 or towards ±1:
- loadings after rotation,
- raw varimax criterion. The common Kaiser variant first divides each row by its communality
In the figure's second example, six tests (three maths, three verbal) are fitted with two factors on a correlation matrix from 300 students. Uniquenesses: Algebra 0.361, Geometry 0.423, Physics 0.433, Reading 0.328, Vocab 0.265, Writing 0.473. The maximum residual is 0.017.
| Test | Unrotated F1 | Unrotated F2 | Varimax verbal | Varimax maths | h² |
|---|---|---|---|---|---|
| Algebra | 0.59 | 0.54 | 0.15 | 0.79 | 0.64 |
| Geometry | 0.56 | 0.52 | 0.14 | 0.75 | 0.58 |
| Physics | 0.61 | 0.44 | 0.23 | 0.72 | 0.57 |
| Reading | 0.77 | −0.29 | 0.79 | 0.22 | 0.67 |
| Vocab | 0.73 | −0.45 | 0.85 | 0.07 | 0.73 |
| Writing | 0.70 | −0.19 | 0.68 | 0.26 | 0.53 |
| SS loadings | 2.64 | 1.08 | 1.91 | 1.81 | 3.72 |
Before rotation the first factor is a "general" factor that every test loads on, and the second is a bipolar maths-versus-verbal contrast. After varimax the same fit reads as two clean, separate abilities. Neither reading is more true than the other, since both reproduce the correlations equally well. Rotation is a choice of coordinates. Oblique rotations such as promax and oblimin go further and allow the factors to correlate, which is often more realistic for abilities or attitudes.
Factor analysis vs PCA
Both methods give a low-rank description of a covariance matrix, and both produce "loadings", so they are often confused. The difference is the noise model. PCA, read probabilistically as probabilistic PCA, is exactly the FA model with isotropic noise . FA gives every variable its own .
- one noise variance per variable, so FA can say one test is noisier than another
- one shared noise variance; the ML solution is the mean of the discarded eigenvalues, and spans the top eigenvectors
| Model | Math | Physics | Literature | Chemistry | Max |residual| | Avg log-lik |
|---|---|---|---|---|---|---|
| FA λ | 0.894 | 0.843 | 0.659 | 0.895 | 0.035 | −4.413 |
| PCA, √(eigenvalue) · u₁ | 0.904 | 0.892 | 0.773 | 0.909 | – | – |
| PPCA W (σ² = 0.321) | 0.855 | 0.843 | 0.731 | 0.860 | 0.144 | −4.528 |
Key differences
- Goal
- PCA keeps the most variance; FA explains the covariance (correlations) and treats unique variance as noise.
- Noise
- PCA/PPCA: one for all variables. FA: a separate per variable.
- Scale
- PCA depends on units: rescale a variable and the components change. FA's ML fit is scale-equivariant: fitting the raw covariance here gives loadings , which are exactly the standardized ones times each column's sd.
- Solution
- PCA has a closed form (one eigendecomposition) and nested components. FA needs iterative fitting, and its 2-factor solution is not the 1-factor solution plus one more column.
- Uniqueness
- PCA components are unique up to sign (with distinct eigenvalues). FA factors are unique only up to rotation.
In code
1import numpy as np23X = np.array([[73,62,83,72],[75,71,79,60],[73,66,70,48],[70,69,67,71],[82,83,77,81],4 [79,75,92,71],[71,52,63,59],[51,52,47,40],[49,58,68,46],[57,58,82,52]], float)5Z = (X - X.mean(0)) / X.std(0) # standardize -> S is the correlation matrix6S = Z.T @ Z / len(Z)78def factor_em(S, k, iters=5000, tol=1e-6):9 p = S.shape[0]10 L = np.full((p, k), 0.6) if k == 1 else np.linalg.eigh(S)[1][:, ::-1][:, :k]11 psi = np.diag(S) - (L ** 2).sum(1)12 for t in range(iters):13 # E-step: beta = (I + L' Psi^-1 L)^-1 L' Psi^-114 LtPi = L.T / psi15 beta = np.linalg.solve(np.eye(k) + LtPi @ L, LtPi)16 Ezz = np.eye(k) - beta @ L + beta @ S @ beta.T # mean of E[zz'|x]17 # M-step18 L_new = S @ beta.T @ np.linalg.inv(Ezz)19 psi = np.diag(S - L_new @ beta @ S)20 done = np.abs(L_new - L).max() < tol21 L = L_new22 if done:23 break24 return L, psi, t + 12526L, psi, n_iter = factor_em(S, k=1)27print(n_iter, L.ravel().round(3), psi.round(3)) # 64 [0.894 0.843 0.659 0.895] [0.201 0.289 0.566 0.199]28print("communalities", (L ** 2).ravel().round(3))29print("residual", np.abs(S - L @ L.T - np.diag(psi)).max().round(3)) # 0.0353031# Factor score of a new student32x = (np.array([80, 70, 60, 75]) - X.mean(0)) / X.std(0)33beta = np.linalg.solve(np.eye(1) + (L.T / psi) @ L, L.T / psi)34print("z_hat", (beta @ x).round(3)) # [0.863]1from sklearn.decomposition import FactorAnalysis2from sklearn.preprocessing import StandardScaler34Z = StandardScaler().fit_transform(X) # population sd, as above5fa = FactorAnalysis(n_components=1, tol=1e-10, max_iter=10000).fit(Z)6print(fa.components_.round(3)) # [[-0.894 -0.843 -0.659 -0.895]] (sign is arbitrary)7print(fa.noise_variance_.round(3)) # [0.201 0.289 0.566 0.199]8print(fa.score(Z)) # average log-likelihood -4.4139print(fa.get_covariance().round(3)) # Lambda Lambda' + Psi1011# Two factors with varimax on a 6-test battery Y (n x 6):12fa2 = FactorAnalysis(n_components=2, rotation="varimax").fit(Y)13loadings = fa2.components_.T # rows = tests, columns = factors14scores = fa2.transform(Y) # posterior means E[z | x]1516# Choosing k: compare held-out log-likelihood17# from sklearn.model_selection import cross_val_score18# [cross_val_score(FactorAnalysis(k), Y).mean() for k in range(1, 4)]Use when
- +You believe a few hidden causes generate the correlations and want to estimate them along with how noisy each variable is.
- +Variables have very different reliabilities: FA down-weights noisy ones instead of letting them shape the components.
- +You want a proper probabilistic model, with a likelihood for model comparison, posterior factor scores with uncertainty, and missing-data handling via EM.
- +Units should not matter: the ML solution is equivariant to rescaling variables.
Avoid when
- −You just want compression or decorrelation: PCA is closed-form, faster, unique and nested.
- −The structure is non-linear or the noise is strongly non-Gaussian; consider ICA, autoencoders or non-linear latent models.
- −Samples are few relative to variables: the fit is unstable and Heywood cases become common.
- −Interpretability is the goal but the rotation is chosen ad hoc; different rotations tell different stories with the same fit.
Where it's used
Psychometrics
The origin of the method (Spearman's g, 1904): intelligence and personality batteries, such as the Big Five traits recovered from hundreds of questionnaire items.
Surveys and marketing
Group Likert items into a few constructs (satisfaction, trust, price sensitivity), check that a scale measures one thing, and drop items with high uniqueness.
Finance
Statistical factor models of asset returns: a few market or sector factors plus idiosyncratic risk give a well-conditioned covariance matrix ΛΛᵀ + Ψ for portfolio optimization.
Covariance estimation
With many variables and few samples, a low-rank-plus-diagonal Σ is far more stable than the raw sample covariance. It is used in genomics, sensor arrays and as a Gaussian-model density.
Check yourself
In a one-factor model on standardized data, two variables have loadings 0.9 and 0.7. What correlation does the model imply between them, and what is the second variable's uniqueness?
Check yourself
What is the essential modelling difference between factor analysis and probabilistic PCA?
Check yourself
You rotate a fitted two-factor solution by 30°. Which of these changes?
Check yourself
An FA fit reports a uniqueness of 0.001 (communality ≈ 1) for one survey item. The most likely interpretation is…
Remember
- ✓Model: , , with diagonal, so .
- ✓Each variance splits into communality and uniqueness . Correlations come only from the shared part.
- ✓EM: the E-step computes , , ; the M-step regresses on the inferred to update , then sets .
- ✓Loadings are identified only up to rotation; varimax picks the rotation with the simplest structure without changing the fit.
- ✓Versus PCA: diagonal instead of isotropic noise, scale-equivariant, models covariance rather than variance, and down-weights noisy variables.
- ✓Choose k with scree plots, parallel analysis, BIC or held-out likelihood, and treat a uniqueness near 0 (a Heywood case) as a warning.