Definition
Independent component analysis (ICA) assumes each observed vector is an unknown linear mix of a few statistically independent, non-Gaussian source signals , and estimates an unmixing matrix so that the outputs are as independent as possible. It recovers the sources up to their order, sign and scale.
At a glance
- Task
- Unsupervised blind source separation / feature extraction
- Input → output
- observations of mixed signals → independent components and an estimated mixing matrix
- Learns
- an unmixing matrix (): a whitening step followed by a rotation
- Objective
- maximize non-Gaussianity (kurtosis or negentropy) of each output — equivalently minimize mutual information, or maximize likelihood
- Training
- whitening , then per FastICA iteration; usually converges in 5–100 iterations
- Prediction
- one matrix–vector product, per sample
- Hyperparameters
n_components, contrast functionfun(logcosh / exp / cube),algorithm(parallel / deflation),whiten,max_iter,tol, random seed- Assumes
- instantaneous linear mixing, independent sources, at most one Gaussian source, at least as many sensors as sources
The problem
Two people talk at once in a room with two microphones. Each microphone records a different weighted sum of the two voices: , . We know neither the voices nor the weights — only the recordings. This is the cocktail-party problem, and the same structure appears whenever sensors pick up several independent processes at once: EEG electrodes record brain activity plus eye blinks plus muscle noise; each pixel of an fMRI scan mixes several brain networks.
The obvious tool is PCA, and it fails. PCA finds orthogonal directions of maximum variance, so it decorrelates the data. But uncorrelated is much weaker than independent: after PCA every rotation of the whitened data is still uncorrelated, and PCA simply picks the rotation aligned with the variance — which has nothing to do with where the sources are. In the worked example below, the PCA components are still and : mixtures. Second-order statistics (covariances) cannot tell the rotations apart; we need higher-order statistics.
The idea
The central limit theorem says a sum of independent random variables is more Gaussian than the variables themselves. So a mixture of the sources is generally more Gaussian than any single source, and it is least Gaussian exactly when it equals one source. ICA turns that around: search over directions for the projection that is as non-Gaussian as possible — that projection is a source.
The search is made easy by whitening first: centre the data and transform it so its covariance is the identity. After that, the remaining unknown is just a rotation (in 2-D, a single angle). ICA is then: rotate until every output axis is maximally non-Gaussian. PCA does the first half (decorrelate); ICA adds the second half (rotate by higher-order statistics).
Vocabulary
- Source sᵢ
- an unobserved, independent signal (a voice, a brain process). Also called an independent component.
- Mixing matrix A
- unknown matrix with ; column is how source shows up in each sensor
- Unmixing matrix W
- the estimate of (or its pseudo-inverse); each row extracts one component
- Statistical independence
- — knowing one source tells you nothing about the others, not even through nonlinear relations. Much stronger than zero correlation.
- Whitening (sphering)
- a linear map with ; uncorrelated, unit-variance coordinates
- Kurtosis
- excess fourth moment ; 0 for a Gaussian, negative for flat (sub-Gaussian: uniform, sine, ±1), positive for peaky/heavy-tailed (super-Gaussian: Laplace, speech)
- Negentropy
- : how much less entropy has than a Gaussian of the same variance; 0 only for a Gaussian
- Contrast function G
- the non-Gaussianity measure being maximized, e.g. ; its derivative is the "nonlinearity" in FastICA
- Deflation / symmetric
- extract components one at a time (orthogonalize against those found), or update all rows of W together and re-orthonormalize
The math
The model. Observations are an instantaneous linear mixture of independent sources, with no noise term:
- one observed sample (e.g. the d microphone readings at one instant)
- the hidden sources at that instant, mutually independent
- unknown mixing matrix of full column rank
- unmixing matrix to be learned
- the estimated independent components
What can never be recovered. If is a solution, so is for any permutation and diagonal scaling (absorb into ). So ICA fixes the scale by convention () and leaves sign and order arbitrary. Worse, if two sources are Gaussian, any rotation of them is again a pair of independent Gaussians, so the model is unidentifiable — ICA allows at most one Gaussian source.
Step 1 — whitening. Centre the data and eigendecompose its covariance:
- sample mean of x
- orthonormal eigenvectors of C (the PCA directions), as columns
- diagonal matrix of eigenvalues (the PCA variances)
- whitening matrix;
- whitened data: uncorrelated, unit variance
Why this helps: , and with unit-variance independent sources , so . The new mixing matrix is orthogonal — a pure rotation (possibly with a reflection). Instead of unknowns in we now search over orthogonal matrices, angles. Keeping only the top eigenvectors in also reduces the dimension when .
Step 2 — why non-Gaussianity finds the sources. Take a unit vector and set with . Because is orthogonal, . Expand the fourth moment for independent, zero-mean, unit-variance sources; every cross term with an odd power vanishes:
- weight of source in the projection ()
- kurtosis of source i (−2 for ±1 signals, −1.5 for a sine, −1.2 for uniform, +3 for Laplace)
On the unit sphere , with equality only when is a coordinate vector . So is maximized when : one source, unmixed. Any genuine mixture shrinks the kurtosis towards 0 — the Gaussian value. For two ±1 sources, a projection at angle from a source has , ranging from (on a source) to (at 45°).
Step 3 — a robust contrast. Sample kurtosis is dominated by a few outliers (fourth powers). FastICA usually maximizes an approximation of negentropy instead:
- a standard Gaussian variable (the reference)
- a slowly growing non-quadratic function; G(u) = u⁴/4 recovers kurtosis
- logcosh steepness, (the alpha entry of fun_args in scikit-learn)
grows only linearly for large , so a single extreme sample cannot dominate; its derivative is bounded. That is why it is the default. The cube/kurtosis contrast is fastest but fragile with outliers; is the most robust and suits strongly super-Gaussian sources.
Step 4 — the FastICA fixed-point update. Maximize subject to . The Lagrange condition is . Its Jacobian is ; because is white, approximate . The Jacobian becomes diagonal and Newton's step , multiplied through by the scalar (we renormalize anyway), simplifies to:
- first and second derivative of : and for logcosh; and for cube
- average over the n samples
- the unnormalized new direction
With the cube nonlinearity, and the update is . In source coordinates this is : each coordinate is cubed, so small coordinates vanish extremely fast — cubic convergence, with no learning rate to tune. A negative kurtosis makes point the "wrong" way; since and give the same component, convergence is tested with .
- all rows updated in parallel, then symmetrically re-orthonormalized so that no two rows converge to the same source
For several components, either run the one-unit update on each row and subtract projections onto earlier rows (deflation, Gram–Schmidt), or update all rows at once and apply the symmetric decorrelation above (parallel, scikit-learn's default — errors do not accumulate from one component to the next). The final unmixing in the original coordinates is and the estimated mixing matrix is its pseudo-inverse.
Worked example: unmixing two ±1 signals
Eight time steps of two independent binary sources (each of the four sign patterns appears twice, so the sample means are 0, variances 1, correlation 0 and both kurtoses are exactly ). They are mixed with , so and . ICA sees only the two columns.
| t | s₁ (hidden) | s₂ (hidden) | x₁ = 2s₁ + s₂ | x₂ = s₁ + s₂ | z₁ | z₂ |
|---|---|---|---|---|---|---|
| 1 | 1 | 1 | 3 | 2 | 1.3764 | 0.3249 |
| 2 | 1 | −1 | 1 | 0 | 0.3249 | −1.3764 |
| 3 | −1 | 1 | −1 | 0 | −0.3249 | 1.3764 |
| 4 | −1 | −1 | −3 | −2 | −1.3764 | −0.3249 |
| 5 | 1 | −1 | 1 | 0 | 0.3249 | −1.3764 |
| 6 | 1 | 1 | 3 | 2 | 1.3764 | 0.3249 |
| 7 | −1 | −1 | −3 | −2 | −1.3764 | −0.3249 |
| 8 | −1 | 1 | −1 | 0 | −0.3249 | 1.3764 |
Whitening by hand
- 1Centre. Both columns already average to 0, so .
- 2Covariance. , , , so (indeed ).
- 3Eigen-decomposition. with eigenvectors , — the PCA axes.
- 4Whitening matrix. . Row 1: .
- 5Check. The columns in the table have mean 0, variance 1, correlation 0. And is a rotation, as the theory promised.
- 6PCA stops here — and is wrong. is still a mixture: its kurtosis is , not . Same for . The sources lie at angles and in the -plane, not on the PCA axes.
FastICA (cube nonlinearity) by hand, starting from the first PCA axis w = (1, 0)
- 1Iteration 1. . and . So , normalized : angle (equivalently ), .
- 2Iteration 2. , so , normalized : angle , .
- 3Iteration 3. , , normalized : angle , .
- 4Iteration 4. is : → converged. The angular error to the true source went : each step cubes (, ).
- 5Second component. In 2-D it is simply the perpendicular .
- 6Back to x-coordinates. , and : the outputs are exactly — the sources, with flipped sign.
- 7New observation. A ninth sample unmixes to . The true sources were : recovered perfectly, up to the unavoidable sign.
| Iter | w (before) | E[z y³] | w⁺ (unnormalized) | new angle | kurt(y) | |w⁺·w| |
|---|---|---|---|---|---|---|
| 1 | (1, 0) | (1.8, 0.4) | (−1.2, 0.4) | 161.57° | −1.8000 | 0.9487 |
| 2 | (−0.9487, 0.3162) | (−1.2649, 0) | (1.5811, −0.9487) | 149.04° | −1.9993 | 0.9762 |
| 3 | (0.8575, −0.5145) | (0.8716, −0.4923) | (−1.7009, 1.0512) | 148.28° | −2.0000 | 0.99991 |
| 4 | (−0.8507, 0.5257) | (−0.8507, 0.5257) | (1.7013, −1.0515) | 148.28° | −2.0000 | 1.000000 |
Notice the shape of the kurtosis curve: it is periodic every 90°, because the two perpendicular outputs are recovered together. With Gaussian sources it is flat up to sampling noise — there is literally no direction to prefer, which is the identifiability condition made visible.
The algorithm, step by step
FastICA (parallel version)
- 1Centre: .
- 2Whiten: eigendecompose (or take an SVD of the data), keep the top components, .
- 3Initialize () randomly and orthonormalize it: .
- 4Update every row: .
- 5Decorrelate: , so the rows stay orthonormal and cannot collapse onto the same source.
- 6Converged? If
tol, stop; otherwise go to the update step (up tomax_iter). - 7Output: unmixing , components , mixing (pseudo-inverse). Each column of is the "pattern" of a source across sensors.
Complexity
- Whitening
- to form the covariance plus for its eigendecomposition (or a truncated SVD for large )
- Per iteration
- for the projections and updates in the whitened -dimensional space, plus for the decorrelation
- Iterations
- few: cubic (kurtosis) or quadratic (logcosh) convergence; scikit-learn's default cap is 200
- Memory
- for the whitened data
Variants and hyperparameters
Hyperparameters and their effect
- n_components
- Number of sources . Whitening keeps the top- PCA directions, discarding low-variance (often noise) directions before ICA. Too few merges sources; too many splits noise into fake components.
- fun (contrast G)
logcosh(default): , robust and general purpose.exp: , the most robust, good for strongly super-Gaussian sources.cube: kurtosis, , fastest but outlier-sensitive.- fun_args alpha
- Steepness of : larger approaches the sign function, so the update weights every sample more equally instead of letting large values dominate.
- algorithm
parallel(default) updates all rows and applies symmetric decorrelation;deflationextracts components one by one — errors accumulate, but you can stop after the first few.- whiten
unit-variance(default) fixes ;arbitrary-varianceleaves the scale free;Falseif your data is already white.- max_iter, tol
- Defaults 200 and 1e-4; convergence is tested on . A ConvergenceWarning usually means near-Gaussian sources or too many components.
- random_state, w_init
- Different starts can reach different local optima on real data — rerun with several seeds and trust only components that reappear.
Other ICA algorithms
- Infomax / extended Infomax
- maximum likelihood with a fixed super-Gaussian (or switching sub/super) source density, optimized by natural-gradient ascent . Standard in EEG toolboxes.
- JADE
- joint diagonalization of fourth-order cumulant matrices; no iteration over samples, good for small d.
- Picard
- a quasi-Newton solver for the Infomax likelihood; faster and more reliable convergence on real data.
- Spatial vs temporal ICA
- in fMRI, samples are voxels and components are independent spatial maps; in EEG, samples are time points and components are independent time courses.
ICA belongs to a family of latent-factor models. The factor analysis article also writes but assumes Gaussian factors — exactly the case ICA cannot rotate — so its loadings are only defined up to rotation. NMF replaces independence with non-negativity. t-SNE and UMAP are nonlinear and for visualization; they do not produce an unmixing matrix.
In code
1import numpy as np23def whiten(X):4 """Center X (n x d) and whiten it with PCA: returns Z (n x d), V, mean."""5 mu = X.mean(axis=0)6 Xc = X - mu7 d, E = np.linalg.eigh(np.cov(Xc, rowvar=False, bias=True))8 order = d.argsort()[::-1]9 d, E = d[order], E[:, order]10 V = np.diag(d ** -0.5) @ E.T # V C V^T = I11 return Xc @ V.T, V, mu1213def sym_decorrelate(W):14 """W <- (W W^T)^(-1/2) W : keeps the rows orthonormal."""15 s, U = np.linalg.eigh(W @ W.T)16 return U @ np.diag(s ** -0.5) @ U.T @ W1718def fastica(X, n_iter=200, tol=1e-6, fun="logcosh", seed=0):19 Z, V, mu = whiten(X)20 n, k = Z.shape21 if fun == "cube":22 g, dg = (lambda u: u ** 3), (lambda u: 3 * u ** 2)23 else: # logcosh: G(u) = log cosh u24 g, dg = np.tanh, (lambda u: 1 - np.tanh(u) ** 2)25 W = sym_decorrelate(np.random.default_rng(seed).normal(size=(k, k)))26 for it in range(n_iter):27 Y = Z @ W.T # n x k projections28 W_new = (g(Y).T @ Z) / n - np.diag(dg(Y).mean(axis=0)) @ W29 W_new = sym_decorrelate(W_new)30 lim = np.max(np.abs(np.abs(np.sum(W_new * W, axis=1)) - 1))31 W = W_new32 if lim < tol:33 break34 unmix = W @ V # y = unmix @ (x - mu)35 return unmix, mu, it + 13637# worked example38s1 = np.array([1, 1, -1, -1, 1, 1, -1, -1.])39s2 = np.array([1, -1, 1, -1, -1, 1, -1, 1.])40A = np.array([[2, 1], [1, 1.]])41X = np.c_[s1, s2] @ A.T42unmix, mu, its = fastica(X, fun="cube")43print("iterations:", its)44print("unmixing W:\n", unmix.round(4))45print("W @ A:\n", (unmix @ A).round(4)) # a signed permutation matrix46print("y for x_new=(3,2):", (unmix @ (np.array([3, 2.]) - mu)).round(4))4748# longer signals: sine + sawtooth, logcosh49t = np.arange(200) / 20050S = np.c_[np.sin(2 * np.pi * 3 * t), 2 * ((5 * t + 0.1) % 1) - 1]51S = (S - S.mean(0)) / S.std(0)52unmix, mu, its = fastica(S @ A.T)53Y = (S @ A.T - mu) @ unmix.T54print("signals: iterations", its, "|corr(y, s)|:\n", np.abs(np.corrcoef(Y.T, S.T)[:2, 2:]).round(3))1import numpy as np2from sklearn.decomposition import FastICA, PCA3from scipy.stats import kurtosis45s1 = np.array([1, 1, -1, -1, 1, 1, -1, -1.])6s2 = np.array([1, -1, 1, -1, -1, 1, -1, 1.])7S = np.c_[s1, s2]8A = np.array([[2, 1], [1, 1.]])9X = S @ A.T # the 8 observed rows1011pca = PCA().fit(X)12print("PCA components:\n", pca.components_.round(4))13print("kurtosis of PCA scores:", kurtosis(pca.transform(X), axis=0).round(3))1415ica = FastICA(n_components=2, fun="cube", whiten="unit-variance", random_state=0, tol=1e-6)16Y = ica.fit_transform(X)17print("iterations:", ica.n_iter_)18print("kurtosis of ICA outputs:", kurtosis(Y, axis=0).round(3))19print("unmixing W:\n", ica.components_.round(4))20print("estimated mixing A:\n", ica.mixing_.round(4))21print("|corr(Y, S)|:\n", np.abs(np.corrcoef(Y.T, S.T)[:2, 2:]).round(3))22print("new x=(3,2) ->", ica.transform([[3, 2]]).round(4))1PCA components:2 [[ 0.8507 0.5257]3 [-0.5257 0.8507]]4kurtosis of PCA scores: [-1.2 -1.2]5iterations: 56kurtosis of ICA outputs: [-2. -2.]7unmixing W:8 [[-1. 1.]9 [-1. 2.]]10estimated mixing A:11 [[-2. 1.]12 [-1. 1.]]13|corr(Y, S)|:14 [[1. 0.]15 [0. 1.]]16new x=(3,2) -> [[-1. 1.]]The library and the hand calculation agree up to the ICA ambiguities: scikit-learn returned , i.e. , , so the new sample maps to instead of our — same sources, different signs. The from-scratch version printed exactly, and on the 200-sample sine + sawtooth it converged in 4 iterations with to each source.
Use when
- +Sensors record linear mixtures of physically independent processes: EEG/MEG, fMRI, audio with several microphones, spectroscopy.
- +You need to remove one specific artifact (eye blinks, line noise, heartbeat) while keeping the rest of the signal.
- +PCA gives components that are uncorrelated but not interpretable, and you suspect non-Gaussian underlying factors.
- +You want sparse, localized features (ICA on natural image patches gives edge-like Gabor filters).
Avoid when
- −The sources are (close to) Gaussian — the problem is unidentifiable and components are arbitrary rotations.
- −Mixing is not instantaneous: echoes, delays and reverberation (convolutive mixtures) need frequency-domain ICA or other methods.
- −You need the actual amplitude, sign or ranking of components — ICA cannot give them.
- −There are fewer sensors than sources (underdetermined), or the mixing changes over time.
- −Your goal is just compression or visualization: PCA, UMAP or t-SNE are simpler and better suited.
Where it's used
EEG / MEG artifact removal
Decompose multichannel recordings, identify the components with eye-blink or muscle topographies, zero them, and remix the clean signal.
fMRI brain networks
Spatial ICA on resting-state scans separates independent networks (default mode, visual, motor) without a task model.
Audio source separation
Separate speakers or instruments recorded by several microphones; in the frequency domain for real rooms.
Finance and telecom
Extract independent factors driving many asset returns; separate users' signals in multi-antenna receivers.
Check yourself
Why does FastICA whiten the data before searching for components?
Check yourself
Two independent ±1 sources (kurtosis −2 each) are combined as y = 0.6 s₁ + 0.8 s₂. What is the kurtosis of y?
Check yourself
Both sources are Gaussian. What does ICA recover?
Check yourself
Two sources with equal kurtosis; the current w is off from a source by an angle whose tangent is 0.5 (26.57°). After one cube-FastICA update the error is about…
Remember
- ✓ICA models with independent, non-Gaussian sources and learns ; sources come back only up to order, sign and scale, and at most one may be Gaussian.
- ✓Whitening (PCA + rescaling) decorrelates the data and leaves only a rotation; PCA stops there and returns mixtures.
- ✓By the central limit theorem mixtures are more Gaussian than sources, so ICA rotates to maximize non-Gaussianity: kurtosis or a logcosh negentropy approximation.
- ✓FastICA is an approximate Newton fixed point plus normalization/decorrelation — no learning rate, cubic or quadratic convergence.
- ✓Check stability: near-Gaussian components, convergence warnings or seed-dependent results mean the components are not meaningful.