ICA

Find a linear unmixing that makes the outputs as statistically independent — as non-Gaussian — as possible.

17 min read1680 wordsadvanced8 viewsupdated 2026-10-07

Definition

Independent component analysis (ICA) assumes each observed vector is an unknown linear mix x=Asx=As of a few statistically independent, non-Gaussian source signals ss, and estimates an unmixing matrix W≈A−1W\approx A^{-1} so that the outputs y=Wxy=Wx are as independent as possible. It recovers the sources up to their order, sign and scale.

At a glance

Task
Unsupervised blind source separation / feature extraction
Input → output
nn observations of dd mixed signals → k≤dk\le d independent components y=W(x−μ)y=W(x-\mu) and an estimated mixing matrix A^\hat A
Learns
an unmixing matrix WW (k×dk\times d): a whitening step VV followed by a rotation
Objective
maximize non-Gaussianity (kurtosis or negentropy) of each output — equivalently minimize mutual information, or maximize likelihood
Training
whitening O(nd2+d3)O(nd^2+d^3), then O(nkd)O(nkd) per FastICA iteration; usually converges in 5–100 iterations
Prediction
one matrix–vector product, O(kd)O(kd) per sample
Hyperparameters
n_components, contrast function fun (logcosh / exp / cube), algorithm (parallel / deflation), whiten, max_iter, tol, random seed
Assumes
instantaneous linear mixing, independent sources, at most one Gaussian source, at least as many sensors as sources

The problem

Two people talk at once in a room with two microphones. Each microphone records a different weighted sum of the two voices: x1=a11s1+a12s2x_1=a_{11}s_1+a_{12}s_2, x2=a21s1+a22s2x_2=a_{21}s_1+a_{22}s_2. We know neither the voices ss nor the weights AA — only the recordings. This is the cocktail-party problem, and the same structure appears whenever sensors pick up several independent processes at once: EEG electrodes record brain activity plus eye blinks plus muscle noise; each pixel of an fMRI scan mixes several brain networks.

The obvious tool is PCA, and it fails. PCA finds orthogonal directions of maximum variance, so it decorrelates the data. But uncorrelated is much weaker than independent: after PCA every rotation of the whitened data is still uncorrelated, and PCA simply picks the rotation aligned with the variance — which has nothing to do with where the sources are. In the worked example below, the PCA components are still 0.85s1+0.53s20.85s_1+0.53s_2 and −0.53s1+0.85s2-0.53s_1+0.85s_2: mixtures. Second-order statistics (covariances) cannot tell the rotations apart; we need higher-order statistics.

The idea

The central limit theorem says a sum of independent random variables is more Gaussian than the variables themselves. So a mixture wopxw^ op x of the sources is generally more Gaussian than any single source, and it is least Gaussian exactly when it equals one source. ICA turns that around: search over directions ww for the projection y=w⊤xy=w^\top x that is as non-Gaussian as possible — that projection is a source.

The search is made easy by whitening first: centre the data and transform it so its covariance is the identity. After that, the remaining unknown is just a rotation (in 2-D, a single angle). ICA is then: rotate until every output axis is maximally non-Gaussian. PCA does the first half (decorrelate); ICA adds the second half (rotate by higher-order statistics).

Vocabulary

Source sᵢ
an unobserved, independent signal (a voice, a brain process). Also called an independent component.
Mixing matrix A
unknown d×kd\times k matrix with x=Asx=As; column aia_i is how source ii shows up in each sensor
Unmixing matrix W
the estimate of A−1A^{-1} (or its pseudo-inverse); each row wiw_i extracts one component yi=wi⊤xy_i=w_i^\top x
Statistical independence
p(s1,…,sk)=∏ipi(si)p(s_1,\dots,s_k)=\prod_i p_i(s_i) — knowing one source tells you nothing about the others, not even through nonlinear relations. Much stronger than zero correlation.
Whitening (sphering)
a linear map VV with Cov⁡(V(x−μ))=I\operatorname{Cov}(V(x-\mu))=I; uncorrelated, unit-variance coordinates zz
Kurtosis
excess fourth moment κ(y)=E[y4]−3(E[y2])2\kappa(y)=E[y^4]-3(E[y^2])^2; 0 for a Gaussian, negative for flat (sub-Gaussian: uniform, sine, ±1), positive for peaky/heavy-tailed (super-Gaussian: Laplace, speech)
Negentropy
J(y)=H(ν)−H(y)≥0J(y)=H(\nu)-H(y)\ge 0: how much less entropy yy has than a Gaussian ν\nu of the same variance; 0 only for a Gaussian
Contrast function G
the non-Gaussianity measure being maximized, e.g. log⁡cosh⁡\log\cosh; its derivative g=G′g=G' is the "nonlinearity" in FastICA
Deflation / symmetric
extract components one at a time (orthogonalize against those found), or update all rows of W together and re-orthonormalize

The math

The model. Observations are an instantaneous linear mixture of independent sources, with no noise term:

x=A s,p(s)=∏i=1kpi(si),y=Wx≈sx = A\,s,\qquad p(s)=\prod_{i=1}^{k}p_i(s_i),\qquad y = W x \approx s
x∈Rdx\in\mathbb{R}^d
one observed sample (e.g. the d microphone readings at one instant)
s∈Rks\in\mathbb{R}^k
the hidden sources at that instant, mutually independent
AA
unknown d×kd\times k mixing matrix of full column rank
WW
k×dk\times d unmixing matrix to be learned
yy
the estimated independent components

What can never be recovered. If ss is a solution, so is PDsPDs for any permutation PP and diagonal scaling DD (absorb D−1P−1D^{-1}P^{-1} into AA). So ICA fixes the scale by convention (E[yi2]=1E[y_i^2]=1) and leaves sign and order arbitrary. Worse, if two sources are Gaussian, any rotation of them is again a pair of independent Gaussians, so the model is unidentifiable — ICA allows at most one Gaussian source.

Step 1 — whitening. Centre the data and eigendecompose its covariance:

C=E[(x−μ)(x−μ)⊤]=E D E⊤,V=D−1/2E⊤,z=V(x−μ)C=E[(x-\mu)(x-\mu)^\top]=E\,D\,E^\top,\qquad V=D^{-1/2}E^\top,\qquad z=V(x-\mu)
μ\mu
sample mean of x
EE
orthonormal eigenvectors of C (the PCA directions), as columns
DD
diagonal matrix of eigenvalues (the PCA variances)
VV
whitening matrix; VCV⊤=IVCV^\top=I
zz
whitened data: uncorrelated, unit variance

Why this helps: z=(VA)sz=(VA)s, and with unit-variance independent sources E[ss⊤]=IE[ss^\top]=I, so I=E[zz⊤]=(VA)(VA)⊤I=E[zz^\top]=(VA)(VA)^\top. The new mixing matrix A~=VA\tilde A=VA is orthogonal — a pure rotation (possibly with a reflection). Instead of d2d^2 unknowns in AA we now search over orthogonal matrices, d(d−1)/2d(d-1)/2 angles. Keeping only the top kk eigenvectors in VV also reduces the dimension when d>kd>k.

Step 2 — why non-Gaussianity finds the sources. Take a unit vector ww and set y=w⊤z=q⊤sy=w^\top z=q^\top s with q=A~⊤wq=\tilde A^\top w. Because A~\tilde A is orthogonal, ∥q∥=1\|q\|=1. Expand the fourth moment for independent, zero-mean, unit-variance sources; every cross term with an odd power vanishes:

E[y4]=∑iqi4E[si4]+6∑i<jqi2qj2,3(E[y2])2=3∑iqi4+6∑i<jqi2qj2E[y^4]=\sum_i q_i^4E[s_i^4]+6\sum_{i<j}q_i^2q_j^2,\qquad 3\big(E[y^2]\big)^2=3\sum_i q_i^4+6\sum_{i<j}q_i^2q_j^2
κ(y)=E[y4]−3(E[y2])2=∑iqi4 κ(si)\kappa(y)=E[y^4]-3\big(E[y^2]\big)^2=\sum_{i}q_i^4\,\kappa(s_i)
qiq_i
weight of source ii in the projection yy (∑iqi2=1\sum_i q_i^2=1)
κ(si)\kappa(s_i)
kurtosis of source i (−2 for ±1 signals, −1.5 for a sine, −1.2 for uniform, +3 for Laplace)

On the unit sphere ∑iqi4≤1\sum_i q_i^4\le 1, with equality only when qq is a coordinate vector ±ei\pm e_i. So ∣κ(y)∣|\kappa(y)| is maximized when y=±siy=\pm s_i: one source, unmixed. Any genuine mixture shrinks the kurtosis towards 0 — the Gaussian value. For two ±1 sources, a projection at angle φ\varphi from a source has κ=−2(cos⁡4φ+sin⁡4φ)=−2+sin⁡22φ\kappa=-2(\cos^4\varphi+\sin^4\varphi)=-2+\sin^2 2\varphi, ranging from −2-2 (on a source) to −1-1 (at 45°).

Step 3 — a robust contrast. Sample kurtosis is dominated by a few outliers (fourth powers). FastICA usually maximizes an approximation of negentropy instead:

J(y)≈(E[G(y)]−E[G(ν)])2,G(u)=1alog⁡cosh⁡(au)  or  G(u)=−e−u2/2J(y)\approx\big(E[G(y)]-E[G(\nu)]\big)^2,\qquad G(u)=\tfrac1a\log\cosh(au)\ \ \text{or}\ \ G(u)=-e^{-u^2/2}
ν\nu
a standard Gaussian variable (the reference)
GG
a slowly growing non-quadratic function; G(u) = u⁴/4 recovers kurtosis
aa
logcosh steepness, 1≤a≤21\le a\le 2 (the alpha entry of fun_args in scikit-learn)

log⁡cosh⁡\log\cosh grows only linearly for large ∣u∣|u|, so a single extreme sample cannot dominate; its derivative g(u)=tanh⁡(au)g(u)=\tanh(au) is bounded. That is why it is the default. The cube/kurtosis contrast is fastest but fragile with outliers; −e−u2/2-e^{-u^2/2} is the most robust and suits strongly super-Gaussian sources.

Step 4 — the FastICA fixed-point update. Maximize E[G(w⊤z)]E[G(w^\top z)] subject to ∥w∥=1\|w\|=1. The Lagrange condition is F(w)=E[z g(w⊤z)]−βw=0F(w)=E[z\,g(w^\top z)]-\beta w=0. Its Jacobian is E[zz⊤g′(w⊤z)]−βIE[zz^\top g'(w^\top z)]-\beta I; because zz is white, approximate E[zz⊤g′(w⊤z)]≈E[zz⊤] E[g′(w⊤z)]=E[g′(w⊤z)] IE[zz^\top g'(w^\top z)]\approx E[zz^\top]\,E[g'(w^\top z)]=E[g'(w^\top z)]\,I. The Jacobian becomes diagonal and Newton's step w←w−F(w)/(E[g′]−β)w\leftarrow w-F(w)/(E[g']-\beta), multiplied through by the scalar β−E[g′]\beta-E[g'] (we renormalize anyway), simplifies to:

w+=E[z g(w⊤z)]−E[g′(w⊤z)] w,w←w+∥w+∥w^{+}=E\big[z\,g(w^\top z)\big]-E\big[g'(w^\top z)\big]\,w,\qquad w\leftarrow\frac{w^{+}}{\|w^{+}\|}
g, g′g,\ g'
first and second derivative of GG: tanh⁡(au)\tanh(au) and a(1−tanh⁡2au)a(1-\tanh^2 au) for logcosh; u3u^3 and 3u23u^2 for cube
E[⋅]E[\cdot]
average over the n samples
w+w^{+}
the unnormalized new direction

With the cube nonlinearity, E[g′(y)]=3E[y2]=3E[g'(y)]=3E[y^2]=3 and the update is w+=E[z(w⊤z)3]−3ww^+=E[z(w^\top z)^3]-3w. In source coordinates this is qi+=κ(si) qi3q_i^+=\kappa(s_i)\,q_i^3: each coordinate is cubed, so small coordinates vanish extremely fast — cubic convergence, with no learning rate to tune. A negative kurtosis makes w+w^+ point the "wrong" way; since ww and −w-w give the same component, convergence is tested with ∣w+⊤w∣→1|w^{+\top}w|\to1.

W←(WW⊤)−1/2 WW\leftarrow (WW^\top)^{-1/2}\,W
WW
all kk rows updated in parallel, then symmetrically re-orthonormalized so that no two rows converge to the same source

For several components, either run the one-unit update on each row and subtract projections onto earlier rows (deflation, Gram–Schmidt), or update all rows at once and apply the symmetric decorrelation above (parallel, scikit-learn's default — errors do not accumulate from one component to the next). The final unmixing in the original coordinates is Wtotal=WVW_{\text{total}}=WV and the estimated mixing matrix is its pseudo-inverse.

Worked example: unmixing two ±1 signals

Eight time steps of two independent binary sources (each of the four sign patterns appears twice, so the sample means are 0, variances 1, correlation 0 and both kurtoses are exactly −2-2). They are mixed with A=(2111)A=\begin{pmatrix}2&1\\1&1\end{pmatrix}, so x1=2s1+s2x_1=2s_1+s_2 and x2=s1+s2x_2=s_1+s_2. ICA sees only the two xx columns.

ts₁ (hidden)s₂ (hidden)x₁ = 2s₁ + s₂x₂ = s₁ + s₂z₁z₂
111321.37640.3249
21−1100.3249−1.3764
3−11−10−0.32491.3764
4−1−1−3−2−1.3764−0.3249
51−1100.3249−1.3764
611321.37640.3249
7−1−1−3−2−1.3764−0.3249
8−11−10−0.32491.3764
The sources (hidden from ICA) and the observed mixtures. Highlighted: rows 1 and 2, used in the hand calculation.

Whitening by hand

  1. 1Centre. Both columns already average to 0, so μ=(0,0)\mu=(0,0).
  2. 2Covariance. E[x12]=9+1+1+9+1+9+9+18=5E[x_1^2]=\tfrac{9+1+1+9+1+9+9+1}{8}=5, E[x1x2]=6+0+0+6+0+6+6+08=3E[x_1x_2]=\tfrac{6+0+0+6+0+6+6+0}{8}=3, E[x22]=168=2E[x_2^2]=\tfrac{16}{8}=2, so C=(5332)C=\begin{pmatrix}5&3\\3&2\end{pmatrix} (indeed AA⊤AA^\top).
  3. 3Eigen-decomposition. λ=7±452=6.8541, 0.1459\lambda=\tfrac{7\pm\sqrt{45}}{2}=6.8541,\ 0.1459 with eigenvectors e1=(0.8507,0.5257)e_1=(0.8507,0.5257), e2=(−0.5257,0.8507)e_2=(-0.5257,0.8507) — the PCA axes.
  4. 4Whitening matrix. V=D−1/2E⊤=(0.8507/6.85410.5257/6.8541−0.5257/0.14590.8507/0.1459)=(0.32490.2008−1.37642.2270)V=D^{-1/2}E^\top=\begin{pmatrix}0.8507/\sqrt{6.8541}&0.5257/\sqrt{6.8541}\\-0.5257/\sqrt{0.1459}&0.8507/\sqrt{0.1459}\end{pmatrix}=\begin{pmatrix}0.3249&0.2008\\-1.3764&2.2270\end{pmatrix}. Row 1: z=V(3,2)⊤=(0.9747+0.4017, −4.1292+4.4540)=(1.3764, 0.3249)z=V(3,2)^\top=(0.9747+0.4017,\ -4.1292+4.4540)=(1.3764,\ 0.3249).
  5. 5Check. The zz columns in the table have mean 0, variance 1, correlation 0. And VA=(0.85070.5257−0.52570.8507)VA=\begin{pmatrix}0.8507&0.5257\\-0.5257&0.8507\end{pmatrix} is a rotation, as the theory promised.
  6. 6PCA stops here — and is wrong. z1=0.8507s1+0.5257s2z_1=0.8507s_1+0.5257s_2 is still a mixture: its kurtosis is −2(0.85074+0.52574)=−1.2-2(0.8507^4+0.5257^4)=-1.2, not −2-2. Same for z2z_2. The sources lie at angles −31.72°-31.72° and 58.28°58.28° in the zz-plane, not on the PCA axes.

FastICA (cube nonlinearity) by hand, starting from the first PCA axis w = (1, 0)

  1. 1Iteration 1. y=w⊤z=z1y=w^\top z=z_1. E[z1y3]=E[z14]=4(1.37644)+4(0.32494)8=1.8E[z_1y^3]=E[z_1^4]=\tfrac{4(1.3764^4)+4(0.3249^4)}{8}=1.8 and E[z2y3]=0.4E[z_2y^3]=0.4. So w+=(1.8,0.4)−3(1,0)=(−1.2, 0.4)w^+=(1.8,0.4)-3(1,0)=(-1.2,\ 0.4), normalized w=(−0.9487, 0.3162)w=(-0.9487,\ 0.3162): angle 161.57°161.57° (equivalently −18.43°-18.43°), κ(y)=−1.80\kappa(y)=-1.80.
  2. 2Iteration 2. E[zy3]=(−1.2649, 0)E[zy^3]=(-1.2649,\ 0), so w+=(−1.2649,0)−3(−0.9487,0.3162)=(1.5811, −0.9487)w^+=(-1.2649,0)-3(-0.9487,0.3162)=(1.5811,\ -0.9487), normalized (0.8575, −0.5145)(0.8575,\ -0.5145): angle 149.04°149.04°, κ=−1.9993\kappa=-1.9993.
  3. 3Iteration 3. E[zy3]=(0.8716, −0.4923)E[zy^3]=(0.8716,\ -0.4923), w+=(−1.7009, 1.0512)w^+=(-1.7009,\ 1.0512), normalized (−0.8507, 0.5257)(-0.8507,\ 0.5257): angle 148.28°148.28°, κ=−2.0000\kappa=-2.0000.
  4. 4Iteration 4. w+w^+ is ±w\pm w: ∣w+⊤w∣=1.000000|w^{+\top}w|=1.000000 → converged. The angular error to the true source went 31.72°→13.29°→0.76°→0°31.72°\to13.29°\to0.76°\to0°: each step cubes tan⁡(error)\tan(\text{error}) (tan⁡31.72°=0.618\tan 31.72°=0.618, 0.6183=0.236=tan⁡13.29°0.618^3=0.236=\tan13.29°).
  5. 5Second component. In 2-D it is simply the perpendicular w2=(−0.5257, −0.8507)w_2=(-0.5257,\ -0.8507).
  6. 6Back to x-coordinates. W=(w1⊤w2⊤)V=(−111−2)W=\begin{pmatrix}w_1^\top\\w_2^\top\end{pmatrix}V=\begin{pmatrix}-1&1\\1&-2\end{pmatrix}, and WA=−IWA=-I: the outputs are exactly y=−sy=-s — the sources, with flipped sign.
  7. 7New observation. A ninth sample x=(3,2)x=(3,2) unmixes to y=Wx=(−3+2, 3−4)=(−1,−1)y=W x=(-3+2,\ 3-4)=(-1,-1). The true sources were (1,1)(1,1): recovered perfectly, up to the unavoidable sign.
Iterw (before)E[z y³]w⁺ (unnormalized)new anglekurt(y)|w⁺·w|
1(1, 0)(1.8, 0.4)(−1.2, 0.4)161.57°−1.80000.9487
2(−0.9487, 0.3162)(−1.2649, 0)(1.5811, −0.9487)149.04°−1.99930.9762
3(0.8575, −0.5145)(0.8716, −0.4923)(−1.7009, 1.0512)148.28°−2.00000.99991
4(−0.8507, 0.5257)(−0.8507, 0.5257)(1.7013, −1.0515)148.28°−2.00001.000000
The FastICA iterates. Angles are mod 180°, because w and −w extract the same component.
mixtures x = Aswhitened z · drag w|kurt(wᵀz)| vs θ- - PCA axesw·· true sources0.511.50°90°180°signals over time · 8 samplesx₁mixturex₂mixturey₁w·z |kurt| 1.20y₂w⊥·z |kurt| 1.20
data:
θ = 0.00° · iterations 0
kurt(y₁) = -1.200 · kurt(y₂) = -1.200
|corr| y₁↔s₁,s₂ = 0.851, 0.526
|corr| y₂↔s₁,s₂ = 0.526, 0.851
w starts on the first PCA axis. Drag it, or press FastICA step: each update jumps towards a peak of the curve.
fig.Left: the 8 mixed rows (dashed violet = PCA axes). Middle: the whitened data, with the unmixing direction w (blue, draggable) and its perpendicular (orange); dotted mint lines are the true sources. Right: |kurtosis| of the projection for every angle — it peaks at 148.28° and 58.28°, where the sources are, and is only 1.2 on the PCA axes. Press FastICA step: w jumps 0° → 161.57° → 149.04° → 148.28° and stops, |corr| with the sources = 1.000. On sine + sawtooth the same three jumps (0° → 158.99° → 148.59° → 148.28°) recover kurtoses −1.500 and −1.202. On two Gaussians the curve never exceeds 0.22 and w wanders without converging.

Notice the shape of the kurtosis curve: it is periodic every 90°, because the two perpendicular outputs are recovered together. With Gaussian sources it is flat up to sampling noise — there is literally no direction to prefer, which is the identifiability condition made visible.

The algorithm, step by step

FastICA (parallel version)

  1. 1Centre: x←x−μx\leftarrow x-\mu.
  2. 2Whiten: eigendecompose C=EDE⊤C=EDE^\top (or take an SVD of the data), keep the top kk components, z=Dk−1/2Ek⊤xz=D_k^{-1/2}E_k^\top x.
  3. 3Initialize WW (k×kk\times k) randomly and orthonormalize it: W←(WW⊤)−1/2WW\leftarrow (WW^\top)^{-1/2}W.
  4. 4Update every row: wi+=E[z g(wi⊤z)]−E[g′(wi⊤z)] wiw_i^+=E[z\,g(w_i^\top z)]-E[g'(w_i^\top z)]\,w_i.
  5. 5Decorrelate: W←(W+W+⊤)−1/2W+W\leftarrow (W^{+}W^{+\top})^{-1/2}W^{+}, so the rows stay orthonormal and cannot collapse onto the same source.
  6. 6Converged? If max⁡i∣∣wi+⊤wi∣−1∣<\max_i\big||w_i^{+\top}w_i|-1\big|< tol, stop; otherwise go to the update step (up to max_iter).
  7. 7Output: unmixing Wtotal=WVW_{\text{total}}=WV, components y=Wtotal(x−μ)y=W_{\text{total}}(x-\mu), mixing A^=Wtotal+\hat A=W_{\text{total}}^{+} (pseudo-inverse). Each column of A^\hat A is the "pattern" of a source across sensors.

Complexity

Whitening
O(nd2)O(nd^2) to form the covariance plus O(d3)O(d^3) for its eigendecomposition (or a truncated SVD for large dd)
Per iteration
O(nk2)O(nk^2) for the projections and updates in the whitened kk-dimensional space, plus O(k3)O(k^3) for the decorrelation
Iterations
few: cubic (kurtosis) or quadratic (logcosh) convergence; scikit-learn's default cap is 200
Memory
O(nk)O(nk) for the whitened data

Variants and hyperparameters

Hyperparameters and their effect

n_components
Number of sources k≤dk\le d. Whitening keeps the top-kk PCA directions, discarding low-variance (often noise) directions before ICA. Too few merges sources; too many splits noise into fake components.
fun (contrast G)
logcosh (default): g=tanh⁡(au)g=\tanh(au), robust and general purpose. exp: g=u e−u2/2g=u\,e^{-u^2/2}, the most robust, good for strongly super-Gaussian sources. cube: kurtosis, g=u3g=u^3, fastest but outlier-sensitive.
fun_args alpha
Steepness a∈[1,2]a\in[1,2] of tanh⁡(au)\tanh(au): larger aa approaches the sign function, so the update weights every sample more equally instead of letting large values dominate.
algorithm
parallel (default) updates all rows and applies symmetric decorrelation; deflation extracts components one by one — errors accumulate, but you can stop after the first few.
whiten
unit-variance (default) fixes E[yi2]=1E[y_i^2]=1; arbitrary-variance leaves the scale free; False if your data is already white.
max_iter, tol
Defaults 200 and 1e-4; convergence is tested on ∣w+⊤w∣→1|w^{+\top}w|\to1. A ConvergenceWarning usually means near-Gaussian sources or too many components.
random_state, w_init
Different starts can reach different local optima on real data — rerun with several seeds and trust only components that reappear.

Other ICA algorithms

Infomax / extended Infomax
maximum likelihood with a fixed super-Gaussian (or switching sub/super) source density, optimized by natural-gradient ascent ΔW∝(I−φ(y)y⊤)W\Delta W\propto(I-\varphi(y)y^\top)W. Standard in EEG toolboxes.
JADE
joint diagonalization of fourth-order cumulant matrices; no iteration over samples, good for small d.
Picard
a quasi-Newton solver for the Infomax likelihood; faster and more reliable convergence on real data.
Spatial vs temporal ICA
in fMRI, samples are voxels and components are independent spatial maps; in EEG, samples are time points and components are independent time courses.

ICA belongs to a family of latent-factor models. The factor analysis article also writes x=Λf+εx=\Lambda f+\varepsilon but assumes Gaussian factors — exactly the case ICA cannot rotate — so its loadings are only defined up to rotation. NMF replaces independence with non-negativity. t-SNE and UMAP are nonlinear and for visualization; they do not produce an unmixing matrix.

In code

python
1import numpy as np
2
3def whiten(X):
4 """Center X (n x d) and whiten it with PCA: returns Z (n x d), V, mean."""
5 mu = X.mean(axis=0)
6 Xc = X - mu
7 d, E = np.linalg.eigh(np.cov(Xc, rowvar=False, bias=True))
8 order = d.argsort()[::-1]
9 d, E = d[order], E[:, order]
10 V = np.diag(d ** -0.5) @ E.T # V C V^T = I
11 return Xc @ V.T, V, mu
12
13def sym_decorrelate(W):
14 """W <- (W W^T)^(-1/2) W : keeps the rows orthonormal."""
15 s, U = np.linalg.eigh(W @ W.T)
16 return U @ np.diag(s ** -0.5) @ U.T @ W
17
18def fastica(X, n_iter=200, tol=1e-6, fun="logcosh", seed=0):
19 Z, V, mu = whiten(X)
20 n, k = Z.shape
21 if fun == "cube":
22 g, dg = (lambda u: u ** 3), (lambda u: 3 * u ** 2)
23 else: # logcosh: G(u) = log cosh u
24 g, dg = np.tanh, (lambda u: 1 - np.tanh(u) ** 2)
25 W = sym_decorrelate(np.random.default_rng(seed).normal(size=(k, k)))
26 for it in range(n_iter):
27 Y = Z @ W.T # n x k projections
28 W_new = (g(Y).T @ Z) / n - np.diag(dg(Y).mean(axis=0)) @ W
29 W_new = sym_decorrelate(W_new)
30 lim = np.max(np.abs(np.abs(np.sum(W_new * W, axis=1)) - 1))
31 W = W_new
32 if lim < tol:
33 break
34 unmix = W @ V # y = unmix @ (x - mu)
35 return unmix, mu, it + 1
36
37# worked example
38s1 = np.array([1, 1, -1, -1, 1, 1, -1, -1.])
39s2 = np.array([1, -1, 1, -1, -1, 1, -1, 1.])
40A = np.array([[2, 1], [1, 1.]])
41X = np.c_[s1, s2] @ A.T
42unmix, mu, its = fastica(X, fun="cube")
43print("iterations:", its)
44print("unmixing W:\n", unmix.round(4))
45print("W @ A:\n", (unmix @ A).round(4)) # a signed permutation matrix
46print("y for x_new=(3,2):", (unmix @ (np.array([3, 2.]) - mu)).round(4))
47
48# longer signals: sine + sawtooth, logcosh
49t = np.arange(200) / 200
50S = np.c_[np.sin(2 * np.pi * 3 * t), 2 * ((5 * t + 0.1) % 1) - 1]
51S = (S - S.mean(0)) / S.std(0)
52unmix, mu, its = fastica(S @ A.T)
53Y = (S @ A.T - mu) @ unmix.T
54print("signals: iterations", its, "|corr(y, s)|:\n", np.abs(np.corrcoef(Y.T, S.T)[:2, 2:]).round(3))
From scratch: PCA whitening + parallel FastICA with logcosh or cube. Prints the worked-example unmixing and separates sine + sawtooth.
python
1import numpy as np
2from sklearn.decomposition import FastICA, PCA
3from scipy.stats import kurtosis
4
5s1 = np.array([1, 1, -1, -1, 1, 1, -1, -1.])
6s2 = np.array([1, -1, 1, -1, -1, 1, -1, 1.])
7S = np.c_[s1, s2]
8A = np.array([[2, 1], [1, 1.]])
9X = S @ A.T # the 8 observed rows
10
11pca = PCA().fit(X)
12print("PCA components:\n", pca.components_.round(4))
13print("kurtosis of PCA scores:", kurtosis(pca.transform(X), axis=0).round(3))
14
15ica = FastICA(n_components=2, fun="cube", whiten="unit-variance", random_state=0, tol=1e-6)
16Y = ica.fit_transform(X)
17print("iterations:", ica.n_iter_)
18print("kurtosis of ICA outputs:", kurtosis(Y, axis=0).round(3))
19print("unmixing W:\n", ica.components_.round(4))
20print("estimated mixing A:\n", ica.mixing_.round(4))
21print("|corr(Y, S)|:\n", np.abs(np.corrcoef(Y.T, S.T)[:2, 2:]).round(3))
22print("new x=(3,2) ->", ica.transform([[3, 2]]).round(4))
scikit-learn on the worked example: PCA leaves kurtosis −1.2, FastICA reaches −2 and recovers the sources up to sign.
text
1PCA components:
2 [[ 0.8507 0.5257]
3 [-0.5257 0.8507]]
4kurtosis of PCA scores: [-1.2 -1.2]
5iterations: 5
6kurtosis of ICA outputs: [-2. -2.]
7unmixing W:
8 [[-1. 1.]
9 [-1. 2.]]
10estimated mixing A:
11 [[-2. 1.]
12 [-1. 1.]]
13|corr(Y, S)|:
14 [[1. 0.]
15 [0. 1.]]
16new x=(3,2) -> [[-1. 1.]]
Output of the scikit-learn script.

The library and the hand calculation agree up to the ICA ambiguities: scikit-learn returned W=(−11−12)W=\begin{pmatrix}-1&1\\-1&2\end{pmatrix}, i.e. y1=−s1y_1=-s_1, y2=+s2y_2=+s_2, so the new sample x=(3,2)x=(3,2) maps to (−1,1)(-1,1) instead of our (−1,−1)(-1,-1) — same sources, different signs. The from-scratch version printed W=(1−1−12)=A−1W=\begin{pmatrix}1&-1\\-1&2\end{pmatrix}=A^{-1} exactly, and on the 200-sample sine + sawtooth it converged in 4 iterations with ∣corr∣=1.000|\text{corr}|=1.000 to each source.

Use when

  • +Sensors record linear mixtures of physically independent processes: EEG/MEG, fMRI, audio with several microphones, spectroscopy.
  • +You need to remove one specific artifact (eye blinks, line noise, heartbeat) while keeping the rest of the signal.
  • +PCA gives components that are uncorrelated but not interpretable, and you suspect non-Gaussian underlying factors.
  • +You want sparse, localized features (ICA on natural image patches gives edge-like Gabor filters).

Avoid when

  • −The sources are (close to) Gaussian — the problem is unidentifiable and components are arbitrary rotations.
  • −Mixing is not instantaneous: echoes, delays and reverberation (convolutive mixtures) need frequency-domain ICA or other methods.
  • −You need the actual amplitude, sign or ranking of components — ICA cannot give them.
  • −There are fewer sensors than sources (underdetermined), or the mixing changes over time.
  • −Your goal is just compression or visualization: PCA, UMAP or t-SNE are simpler and better suited.

Where it's used

EEG / MEG artifact removal

Decompose multichannel recordings, identify the components with eye-blink or muscle topographies, zero them, and remix the clean signal.

fMRI brain networks

Spatial ICA on resting-state scans separates independent networks (default mode, visual, motor) without a task model.

Audio source separation

Separate speakers or instruments recorded by several microphones; in the frequency domain for real rooms.

Finance and telecom

Extract independent factors driving many asset returns; separate users' signals in multi-antenna receivers.

Check yourself

Why does FastICA whiten the data before searching for components?

Check yourself

Two independent ±1 sources (kurtosis −2 each) are combined as y = 0.6 s₁ + 0.8 s₂. What is the kurtosis of y?

Check yourself

Both sources are Gaussian. What does ICA recover?

Check yourself

Two sources with equal kurtosis; the current w is off from a source by an angle whose tangent is 0.5 (26.57°). After one cube-FastICA update the error is about…

Remember

  • ✓ICA models x=Asx=As with independent, non-Gaussian sources and learns W≈A−1W\approx A^{-1}; sources come back only up to order, sign and scale, and at most one may be Gaussian.
  • ✓Whitening (PCA + rescaling) decorrelates the data and leaves only a rotation; PCA stops there and returns mixtures.
  • ✓By the central limit theorem mixtures are more Gaussian than sources, so ICA rotates to maximize non-Gaussianity: kurtosis ∑iqi4κi\sum_i q_i^4\kappa_i or a logcosh negentropy approximation.
  • ✓FastICA is an approximate Newton fixed point w+=E[zg(w⊤z)]−E[g′(w⊤z)]ww^+=E[zg(w^\top z)]-E[g'(w^\top z)]w plus normalization/decorrelation — no learning rate, cubic or quadratic convergence.
  • ✓Check stability: near-Gaussian components, convergence warnings or seed-dependent results mean the components are not meaningful.
← back to ML Algorithms roadmap