# ICA

> Find a linear unmixing that makes the outputs as statistically independent — as non-Gaussian — as possible.

Source: https://www.mlfoundry.in/learn/ica  
Section: ML Algorithms › Unsupervised · Dimensionality Reduction  
Level: advanced · 17 min read · updated 2026-10-07

**Independent component analysis (ICA):** assumes each observed vector is an unknown linear mix $x=As$ of a few **statistically independent, non-Gaussian** source signals $s$, and estimates an unmixing matrix $W\approx A^{-1}$ so that the outputs $y=Wx$ are as independent as possible. It recovers the sources up to their order, sign and scale.

- **Task:** Unsupervised blind source separation / feature extraction
- **Input → output:** $n$ observations of $d$ mixed signals → $k\le d$ independent components $y=W(x-\mu)$ and an estimated mixing matrix $\hat A$
- **Learns:** an unmixing matrix $W$ ($k\times d$): a whitening step $V$ followed by a rotation
- **Objective:** maximize non-Gaussianity (kurtosis or negentropy) of each output — equivalently minimize mutual information, or maximize likelihood
- **Training:** whitening $O(nd^2+d^3)$, then $O(nkd)$ per FastICA iteration; usually converges in 5–100 iterations
- **Prediction:** one matrix–vector product, $O(kd)$ per sample
- **Hyperparameters:** `n_components`, contrast function `fun` (logcosh / exp / cube), `algorithm` (parallel / deflation), `whiten`, `max_iter`, `tol`, random seed
- **Assumes:** instantaneous linear mixing, independent sources, **at most one Gaussian source**, at least as many sensors as sources

## The problem

Two people talk at once in a room with two microphones. Each microphone records a different weighted sum of the two voices: $x_1=a_{11}s_1+a_{12}s_2$, $x_2=a_{21}s_1+a_{22}s_2$. We know neither the voices $s$ nor the weights $A$ — only the recordings. This is the **cocktail-party problem**, and the same structure appears whenever sensors pick up several independent processes at once: EEG electrodes record brain activity plus eye blinks plus muscle noise; each pixel of an fMRI scan mixes several brain networks.

The obvious tool is **PCA**, and it fails. PCA finds orthogonal directions of maximum variance, so it **decorrelates** the data. But uncorrelated is much weaker than independent: after PCA every rotation of the whitened data is still uncorrelated, and PCA simply picks the rotation aligned with the variance — which has nothing to do with where the sources are. In the worked example below, the PCA components are still $0.85s_1+0.53s_2$ and $-0.53s_1+0.85s_2$: mixtures. Second-order statistics (covariances) cannot tell the rotations apart; we need higher-order statistics.

## The idea

The **central limit theorem** says a sum of independent random variables is *more Gaussian* than the variables themselves. So a mixture $w^	op x$ of the sources is generally more Gaussian than any single source, and it is *least* Gaussian exactly when it equals one source. ICA turns that around: search over directions $w$ for the projection $y=w^\top x$ that is **as non-Gaussian as possible** — that projection is a source.

The search is made easy by **whitening** first: centre the data and transform it so its covariance is the identity. After that, the remaining unknown is just a **rotation** (in 2-D, a single angle). ICA is then: rotate until every output axis is maximally non-Gaussian. PCA does the first half (decorrelate); ICA adds the second half (rotate by higher-order statistics).

**Vocabulary**

- **Source sᵢ:** an unobserved, independent signal (a voice, a brain process). Also called an *independent component*.
- **Mixing matrix A:** unknown $d\times k$ matrix with $x=As$; column $a_i$ is how source $i$ shows up in each sensor
- **Unmixing matrix W:** the estimate of $A^{-1}$ (or its pseudo-inverse); each row $w_i$ extracts one component $y_i=w_i^\top x$
- **Statistical independence:** $p(s_1,\dots,s_k)=\prod_i p_i(s_i)$ — knowing one source tells you nothing about the others, not even through nonlinear relations. Much stronger than zero correlation.
- **Whitening (sphering):** a linear map $V$ with $\operatorname{Cov}(V(x-\mu))=I$; uncorrelated, unit-variance coordinates $z$
- **Kurtosis:** excess fourth moment $\kappa(y)=E[y^4]-3(E[y^2])^2$; 0 for a Gaussian, negative for flat (**sub-Gaussian**: uniform, sine, ±1), positive for peaky/heavy-tailed (**super-Gaussian**: Laplace, speech)
- **Negentropy:** $J(y)=H(\nu)-H(y)\ge 0$: how much less entropy $y$ has than a Gaussian $\nu$ of the same variance; 0 only for a Gaussian
- **Contrast function G:** the non-Gaussianity measure being maximized, e.g. $\log\cosh$; its derivative $g=G'$ is the "nonlinearity" in FastICA
- **Deflation / symmetric:** extract components one at a time (orthogonalize against those found), or update all rows of W together and re-orthonormalize

## The math

**The model.** Observations are an instantaneous linear mixture of independent sources, with no noise term:

$$
x = A\,s,\qquad p(s)=\prod_{i=1}^{k}p_i(s_i),\qquad y = W x \approx s
$$

- $x\in\mathbb{R}^d$: one observed sample (e.g. the d microphone readings at one instant)
- $s\in\mathbb{R}^k$: the hidden sources at that instant, mutually independent
- $A$: unknown $d\times k$ mixing matrix of full column rank
- $W$: $k\times d$ unmixing matrix to be learned
- $y$: the estimated independent components

**What can never be recovered.** If $s$ is a solution, so is $PDs$ for any permutation $P$ and diagonal scaling $D$ (absorb $D^{-1}P^{-1}$ into $A$). So ICA fixes the scale by convention ($E[y_i^2]=1$) and leaves **sign and order arbitrary**. Worse, if two sources are Gaussian, any rotation of them is again a pair of independent Gaussians, so the model is unidentifiable — ICA allows **at most one Gaussian source**.

**Step 1 — whitening.** Centre the data and eigendecompose its covariance:

$$
C=E[(x-\mu)(x-\mu)^\top]=E\,D\,E^\top,\qquad V=D^{-1/2}E^\top,\qquad z=V(x-\mu)
$$

- $\mu$: sample mean of x
- $E$: orthonormal eigenvectors of C (the PCA directions), as columns
- $D$: diagonal matrix of eigenvalues (the PCA variances)
- $V$: whitening matrix; $VCV^\top=I$
- $z$: whitened data: uncorrelated, unit variance

Why this helps: $z=(VA)s$, and with unit-variance independent sources $E[ss^\top]=I$, so $I=E[zz^\top]=(VA)(VA)^\top$. The new mixing matrix $\tilde A=VA$ is **orthogonal** — a pure rotation (possibly with a reflection). Instead of $d^2$ unknowns in $A$ we now search over orthogonal matrices, $d(d-1)/2$ angles. Keeping only the top $k$ eigenvectors in $V$ also reduces the dimension when $d>k$.

**Step 2 — why non-Gaussianity finds the sources.** Take a unit vector $w$ and set $y=w^\top z=q^\top s$ with $q=\tilde A^\top w$. Because $\tilde A$ is orthogonal, $\|q\|=1$. Expand the fourth moment for independent, zero-mean, unit-variance sources; every cross term with an odd power vanishes:

$$
E[y^4]=\sum_i q_i^4E[s_i^4]+6\sum_{i<j}q_i^2q_j^2,\qquad 3\big(E[y^2]\big)^2=3\sum_i q_i^4+6\sum_{i<j}q_i^2q_j^2
$$

$$
\kappa(y)=E[y^4]-3\big(E[y^2]\big)^2=\sum_{i}q_i^4\,\kappa(s_i)
$$

- $q_i$: weight of source $i$ in the projection $y$ ($\sum_i q_i^2=1$)
- $\kappa(s_i)$: kurtosis of source i (−2 for ±1 signals, −1.5 for a sine, −1.2 for uniform, +3 for Laplace)

On the unit sphere $\sum_i q_i^4\le 1$, with equality only when $q$ is a coordinate vector $\pm e_i$. So $|\kappa(y)|$ is maximized when $y=\pm s_i$: **one source, unmixed**. Any genuine mixture shrinks the kurtosis towards 0 — the Gaussian value. For two ±1 sources, a projection at angle $\varphi$ from a source has $\kappa=-2(\cos^4\varphi+\sin^4\varphi)=-2+\sin^2 2\varphi$, ranging from $-2$ (on a source) to $-1$ (at 45°).

**Step 3 — a robust contrast.** Sample kurtosis is dominated by a few outliers (fourth powers). FastICA usually maximizes an approximation of **negentropy** instead:

$$
J(y)\approx\big(E[G(y)]-E[G(\nu)]\big)^2,\qquad G(u)=\tfrac1a\log\cosh(au)\ \ \text{or}\ \ G(u)=-e^{-u^2/2}
$$

- $\nu$: a standard Gaussian variable (the reference)
- $G$: a slowly growing non-quadratic function; G(u) = u⁴/4 recovers kurtosis
- $a$: logcosh steepness, $1\le a\le 2$ (the alpha entry of fun_args in scikit-learn)

$\log\cosh$ grows only linearly for large $|u|$, so a single extreme sample cannot dominate; its derivative $g(u)=\tanh(au)$ is bounded. That is why it is the default. The cube/kurtosis contrast is fastest but fragile with outliers; $-e^{-u^2/2}$ is the most robust and suits strongly super-Gaussian sources.

**Step 4 — the FastICA fixed-point update.** Maximize $E[G(w^\top z)]$ subject to $\|w\|=1$. The Lagrange condition is $F(w)=E[z\,g(w^\top z)]-\beta w=0$. Its Jacobian is $E[zz^\top g'(w^\top z)]-\beta I$; because $z$ is white, approximate $E[zz^\top g'(w^\top z)]\approx E[zz^\top]\,E[g'(w^\top z)]=E[g'(w^\top z)]\,I$. The Jacobian becomes diagonal and Newton's step $w\leftarrow w-F(w)/(E[g']-\beta)$, multiplied through by the scalar $\beta-E[g']$ (we renormalize anyway), simplifies to:

$$
w^{+}=E\big[z\,g(w^\top z)\big]-E\big[g'(w^\top z)\big]\,w,\qquad w\leftarrow\frac{w^{+}}{\|w^{+}\|}
$$

- $g,\ g'$: first and second derivative of $G$: $\tanh(au)$ and $a(1-\tanh^2 au)$ for logcosh; $u^3$ and $3u^2$ for cube
- $E[\cdot]$: average over the n samples
- $w^{+}$: the unnormalized new direction

With the cube nonlinearity, $E[g'(y)]=3E[y^2]=3$ and the update is $w^+=E[z(w^\top z)^3]-3w$. In source coordinates this is $q_i^+=\kappa(s_i)\,q_i^3$: each coordinate is **cubed**, so small coordinates vanish extremely fast — **cubic convergence**, with no learning rate to tune. A negative kurtosis makes $w^+$ point the "wrong" way; since $w$ and $-w$ give the same component, convergence is tested with $|w^{+\top}w|\to1$.

$$
W\leftarrow (WW^\top)^{-1/2}\,W
$$

- $W$: all $k$ rows updated in parallel, then symmetrically re-orthonormalized so that no two rows converge to the same source

For several components, either run the one-unit update on each row and subtract projections onto earlier rows (**deflation**, Gram–Schmidt), or update all rows at once and apply the symmetric decorrelation above (**parallel**, scikit-learn's default — errors do not accumulate from one component to the next). The final unmixing in the original coordinates is $W_{\text{total}}=WV$ and the estimated mixing matrix is its pseudo-inverse.

> **Three objectives, one answer:** Maximizing non-Gaussianity, minimizing the mutual information $I(y_1,\dots,y_k)$ between outputs, and maximum likelihood of $x=As$ with super-Gaussian source priors (*Infomax*) lead to almost the same estimating equations. Under whitening, $I(y)=\text{const}-\sum_i J(y_i)$: making each output as non-Gaussian as possible *is* making the outputs as independent as possible.

## Worked example: unmixing two ±1 signals

Eight time steps of two independent binary sources (each of the four sign patterns appears twice, so the sample means are 0, variances 1, correlation 0 and both kurtoses are exactly $-2$). They are mixed with $A=\begin{pmatrix}2&1\\1&1\end{pmatrix}$, so $x_1=2s_1+s_2$ and $x_2=s_1+s_2$. ICA sees only the two $x$ columns.

The sources (hidden from ICA) and the observed mixtures. Highlighted: rows 1 and 2, used in the hand calculation.

| t | s₁ (hidden) | s₂ (hidden) | x₁ = 2s₁ + s₂ | x₂ = s₁ + s₂ | z₁ | z₂ |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 1 | 1 | 3 | 2 | 1.3764 | 0.3249 |
| 2 | 1 | −1 | 1 | 0 | 0.3249 | −1.3764 |
| 3 | −1 | 1 | −1 | 0 | −0.3249 | 1.3764 |
| 4 | −1 | −1 | −3 | −2 | −1.3764 | −0.3249 |
| 5 | 1 | −1 | 1 | 0 | 0.3249 | −1.3764 |
| 6 | 1 | 1 | 3 | 2 | 1.3764 | 0.3249 |
| 7 | −1 | −1 | −3 | −2 | −1.3764 | −0.3249 |
| 8 | −1 | 1 | −1 | 0 | −0.3249 | 1.3764 |

**Whitening by hand**

1. **Centre.** Both columns already average to 0, so $\mu=(0,0)$.
2. **Covariance.** $E[x_1^2]=\tfrac{9+1+1+9+1+9+9+1}{8}=5$, $E[x_1x_2]=\tfrac{6+0+0+6+0+6+6+0}{8}=3$, $E[x_2^2]=\tfrac{16}{8}=2$, so $C=\begin{pmatrix}5&3\\3&2\end{pmatrix}$ (indeed $AA^\top$).
3. **Eigen-decomposition.** $\lambda=\tfrac{7\pm\sqrt{45}}{2}=6.8541,\ 0.1459$ with eigenvectors $e_1=(0.8507,0.5257)$, $e_2=(-0.5257,0.8507)$ — the PCA axes.
4. **Whitening matrix.** $V=D^{-1/2}E^\top=\begin{pmatrix}0.8507/\sqrt{6.8541}&0.5257/\sqrt{6.8541}\\-0.5257/\sqrt{0.1459}&0.8507/\sqrt{0.1459}\end{pmatrix}=\begin{pmatrix}0.3249&0.2008\\-1.3764&2.2270\end{pmatrix}$. Row 1: $z=V(3,2)^\top=(0.9747+0.4017,\ -4.1292+4.4540)=(1.3764,\ 0.3249)$.
5. **Check.** The $z$ columns in the table have mean 0, variance 1, correlation 0. And $VA=\begin{pmatrix}0.8507&0.5257\\-0.5257&0.8507\end{pmatrix}$ is a rotation, as the theory promised.
6. **PCA stops here — and is wrong.** $z_1=0.8507s_1+0.5257s_2$ is still a mixture: its kurtosis is $-2(0.8507^4+0.5257^4)=-1.2$, not $-2$. Same for $z_2$. The sources lie at angles $-31.72°$ and $58.28°$ in the $z$-plane, not on the PCA axes.

**FastICA (cube nonlinearity) by hand, starting from the first PCA axis w = (1, 0)**

1. **Iteration 1.** $y=w^\top z=z_1$. $E[z_1y^3]=E[z_1^4]=\tfrac{4(1.3764^4)+4(0.3249^4)}{8}=1.8$ and $E[z_2y^3]=0.4$. So $w^+=(1.8,0.4)-3(1,0)=(-1.2,\ 0.4)$, normalized $w=(-0.9487,\ 0.3162)$: angle $161.57°$ (equivalently $-18.43°$), $\kappa(y)=-1.80$.
2. **Iteration 2.** $E[zy^3]=(-1.2649,\ 0)$, so $w^+=(-1.2649,0)-3(-0.9487,0.3162)=(1.5811,\ -0.9487)$, normalized $(0.8575,\ -0.5145)$: angle $149.04°$, $\kappa=-1.9993$.
3. **Iteration 3.** $E[zy^3]=(0.8716,\ -0.4923)$, $w^+=(-1.7009,\ 1.0512)$, normalized $(-0.8507,\ 0.5257)$: angle $148.28°$, $\kappa=-2.0000$.
4. **Iteration 4.** $w^+$ is $\pm w$: $|w^{+\top}w|=1.000000$ → **converged**. The angular error to the true source went $31.72°\to13.29°\to0.76°\to0°$: each step cubes $\tan(\text{error})$ ($\tan 31.72°=0.618$, $0.618^3=0.236=\tan13.29°$).
5. **Second component.** In 2-D it is simply the perpendicular $w_2=(-0.5257,\ -0.8507)$.
6. **Back to x-coordinates.** $W=\begin{pmatrix}w_1^\top\\w_2^\top\end{pmatrix}V=\begin{pmatrix}-1&1\\1&-2\end{pmatrix}$, and $WA=-I$: the outputs are exactly $y=-s$ — the sources, with flipped sign.
7. **New observation.** A ninth sample $x=(3,2)$ unmixes to $y=W x=(-3+2,\ 3-4)=(-1,-1)$. The true sources were $(1,1)$: recovered perfectly, up to the unavoidable sign.

The FastICA iterates. Angles are mod 180°, because w and −w extract the same component.

| Iter | w (before) | E[z y³] | w⁺ (unnormalized) | new angle | kurt(y) | \|w⁺·w\| |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | (1, 0) | (1.8, 0.4) | (−1.2, 0.4) | 161.57° | −1.8000 | 0.9487 |
| 2 | (−0.9487, 0.3162) | (−1.2649, 0) | (1.5811, −0.9487) | 149.04° | −1.9993 | 0.9762 |
| 3 | (0.8575, −0.5145) | (0.8716, −0.4923) | (−1.7009, 1.0512) | 148.28° | −2.0000 | 0.99991 |
| 4 | (−0.8507, 0.5257) | (−0.8507, 0.5257) | (1.7013, −1.0515) | 148.28° | −2.0000 | 1.000000 |

> Figure: Left: the 8 mixed rows (dashed violet = PCA axes). Middle: the whitened data, with the unmixing direction w (blue, draggable) and its perpendicular (orange); dotted mint lines are the true sources. Right: |kurtosis| of the projection for every angle — it peaks at 148.28° and 58.28°, where the sources are, and is only 1.2 on the PCA axes. Press FastICA step: w jumps 0° → 161.57° → 149.04° → 148.28° and stops, |corr| with the sources = 1.000. On sine + sawtooth the same three jumps (0° → 158.99° → 148.59° → 148.28°) recover kurtoses −1.500 and −1.202. On two Gaussians the curve never exceeds 0.22 and w wanders without converging.

Notice the shape of the kurtosis curve: it is periodic every 90°, because the two perpendicular outputs are recovered together. With Gaussian sources it is flat up to sampling noise — there is literally no direction to prefer, which is the identifiability condition made visible.

## The algorithm, step by step

**FastICA (parallel version)**

1. **Centre:** $x\leftarrow x-\mu$.
2. **Whiten:** eigendecompose $C=EDE^\top$ (or take an SVD of the data), keep the top $k$ components, $z=D_k^{-1/2}E_k^\top x$.
3. **Initialize** $W$ ($k\times k$) randomly and orthonormalize it: $W\leftarrow (WW^\top)^{-1/2}W$.
4. **Update every row:** $w_i^+=E[z\,g(w_i^\top z)]-E[g'(w_i^\top z)]\,w_i$.
5. **Decorrelate:** $W\leftarrow (W^{+}W^{+\top})^{-1/2}W^{+}$, so the rows stay orthonormal and cannot collapse onto the same source.
6. **Converged?** If $\max_i\big||w_i^{+\top}w_i|-1\big|<$ `tol`, stop; otherwise go to the update step (up to `max_iter`).
7. **Output:** unmixing $W_{\text{total}}=WV$, components $y=W_{\text{total}}(x-\mu)$, mixing $\hat A=W_{\text{total}}^{+}$ (pseudo-inverse). Each column of $\hat A$ is the "pattern" of a source across sensors.

**Complexity**

- **Whitening:** $O(nd^2)$ to form the covariance plus $O(d^3)$ for its eigendecomposition (or a truncated SVD for large $d$)
- **Per iteration:** $O(nk^2)$ for the projections and updates in the whitened $k$-dimensional space, plus $O(k^3)$ for the decorrelation
- **Iterations:** few: cubic (kurtosis) or quadratic (logcosh) convergence; scikit-learn's default cap is 200
- **Memory:** $O(nk)$ for the whitened data

## Variants and hyperparameters

**Hyperparameters and their effect**

- **n_components:** Number of sources $k\le d$. Whitening keeps the top-$k$ PCA directions, discarding low-variance (often noise) directions before ICA. Too few merges sources; too many splits noise into fake components.
- **fun (contrast G):** `logcosh` (default): $g=\tanh(au)$, robust and general purpose. `exp`: $g=u\,e^{-u^2/2}$, the most robust, good for strongly super-Gaussian sources. `cube`: kurtosis, $g=u^3$, fastest but outlier-sensitive.
- **fun_args alpha:** Steepness $a\in[1,2]$ of $\tanh(au)$: larger $a$ approaches the sign function, so the update weights every sample more equally instead of letting large values dominate.
- **algorithm:** `parallel` (default) updates all rows and applies symmetric decorrelation; `deflation` extracts components one by one — errors accumulate, but you can stop after the first few.
- **whiten:** `unit-variance` (default) fixes $E[y_i^2]=1$; `arbitrary-variance` leaves the scale free; `False` if your data is already white.
- **max_iter, tol:** Defaults 200 and 1e-4; convergence is tested on $|w^{+\top}w|\to1$. A ConvergenceWarning usually means near-Gaussian sources or too many components.
- **random_state, w_init:** Different starts can reach different local optima on real data — rerun with several seeds and trust only components that reappear.

**Other ICA algorithms**

- **Infomax / extended Infomax:** maximum likelihood with a fixed super-Gaussian (or switching sub/super) source density, optimized by natural-gradient ascent $\Delta W\propto(I-\varphi(y)y^\top)W$. Standard in EEG toolboxes.
- **JADE:** joint diagonalization of fourth-order cumulant matrices; no iteration over samples, good for small d.
- **Picard:** a quasi-Newton solver for the Infomax likelihood; faster and more reliable convergence on real data.
- **Spatial vs temporal ICA:** in fMRI, samples are voxels and components are independent spatial maps; in EEG, samples are time points and components are independent time courses.

ICA belongs to a family of latent-factor models. The **factor analysis** article also writes $x=\Lambda f+\varepsilon$ but assumes Gaussian factors — exactly the case ICA cannot rotate — so its loadings are only defined up to rotation. **NMF** replaces independence with non-negativity. **t-SNE** and **UMAP** are nonlinear and for visualization; they do not produce an unmixing matrix.

## In code

```python
import numpy as np

def whiten(X):
    """Center X (n x d) and whiten it with PCA: returns Z (n x d), V, mean."""
    mu = X.mean(axis=0)
    Xc = X - mu
    d, E = np.linalg.eigh(np.cov(Xc, rowvar=False, bias=True))
    order = d.argsort()[::-1]
    d, E = d[order], E[:, order]
    V = np.diag(d ** -0.5) @ E.T                 # V C V^T = I
    return Xc @ V.T, V, mu

def sym_decorrelate(W):
    """W <- (W W^T)^(-1/2) W : keeps the rows orthonormal."""
    s, U = np.linalg.eigh(W @ W.T)
    return U @ np.diag(s ** -0.5) @ U.T @ W

def fastica(X, n_iter=200, tol=1e-6, fun="logcosh", seed=0):
    Z, V, mu = whiten(X)
    n, k = Z.shape
    if fun == "cube":
        g, dg = (lambda u: u ** 3), (lambda u: 3 * u ** 2)
    else:                                       # logcosh: G(u) = log cosh u
        g, dg = np.tanh, (lambda u: 1 - np.tanh(u) ** 2)
    W = sym_decorrelate(np.random.default_rng(seed).normal(size=(k, k)))
    for it in range(n_iter):
        Y = Z @ W.T                              # n x k projections
        W_new = (g(Y).T @ Z) / n - np.diag(dg(Y).mean(axis=0)) @ W
        W_new = sym_decorrelate(W_new)
        lim = np.max(np.abs(np.abs(np.sum(W_new * W, axis=1)) - 1))
        W = W_new
        if lim < tol:
            break
    unmix = W @ V                                # y = unmix @ (x - mu)
    return unmix, mu, it + 1

# worked example
s1 = np.array([1, 1, -1, -1, 1, 1, -1, -1.])
s2 = np.array([1, -1, 1, -1, -1, 1, -1, 1.])
A = np.array([[2, 1], [1, 1.]])
X = np.c_[s1, s2] @ A.T
unmix, mu, its = fastica(X, fun="cube")
print("iterations:", its)
print("unmixing W:\n", unmix.round(4))
print("W @ A:\n", (unmix @ A).round(4))           # a signed permutation matrix
print("y for x_new=(3,2):", (unmix @ (np.array([3, 2.]) - mu)).round(4))

# longer signals: sine + sawtooth, logcosh
t = np.arange(200) / 200
S = np.c_[np.sin(2 * np.pi * 3 * t), 2 * ((5 * t + 0.1) % 1) - 1]
S = (S - S.mean(0)) / S.std(0)
unmix, mu, its = fastica(S @ A.T)
Y = (S @ A.T - mu) @ unmix.T
print("signals: iterations", its, "|corr(y, s)|:\n", np.abs(np.corrcoef(Y.T, S.T)[:2, 2:]).round(3))
```

From scratch: PCA whitening + parallel FastICA with logcosh or cube. Prints the worked-example unmixing and separates sine + sawtooth.

```python
import numpy as np
from sklearn.decomposition import FastICA, PCA
from scipy.stats import kurtosis

s1 = np.array([1, 1, -1, -1, 1, 1, -1, -1.])
s2 = np.array([1, -1, 1, -1, -1, 1, -1, 1.])
S = np.c_[s1, s2]
A = np.array([[2, 1], [1, 1.]])
X = S @ A.T                                   # the 8 observed rows

pca = PCA().fit(X)
print("PCA components:\n", pca.components_.round(4))
print("kurtosis of PCA scores:", kurtosis(pca.transform(X), axis=0).round(3))

ica = FastICA(n_components=2, fun="cube", whiten="unit-variance", random_state=0, tol=1e-6)
Y = ica.fit_transform(X)
print("iterations:", ica.n_iter_)
print("kurtosis of ICA outputs:", kurtosis(Y, axis=0).round(3))
print("unmixing W:\n", ica.components_.round(4))
print("estimated mixing A:\n", ica.mixing_.round(4))
print("|corr(Y, S)|:\n", np.abs(np.corrcoef(Y.T, S.T)[:2, 2:]).round(3))
print("new x=(3,2) ->", ica.transform([[3, 2]]).round(4))
```

scikit-learn on the worked example: PCA leaves kurtosis −1.2, FastICA reaches −2 and recovers the sources up to sign.

```text
PCA components:
 [[ 0.8507  0.5257]
 [-0.5257  0.8507]]
kurtosis of PCA scores: [-1.2 -1.2]
iterations: 5
kurtosis of ICA outputs: [-2. -2.]
unmixing W:
 [[-1.  1.]
 [-1.  2.]]
estimated mixing A:
 [[-2.  1.]
 [-1.  1.]]
|corr(Y, S)|:
 [[1. 0.]
 [0. 1.]]
new x=(3,2) -> [[-1.  1.]]
```

Output of the scikit-learn script.

The library and the hand calculation agree up to the ICA ambiguities: scikit-learn returned $W=\begin{pmatrix}-1&1\\-1&2\end{pmatrix}$, i.e. $y_1=-s_1$, $y_2=+s_2$, so the new sample $x=(3,2)$ maps to $(-1,1)$ instead of our $(-1,-1)$ — same sources, different signs. The from-scratch version printed $W=\begin{pmatrix}1&-1\\-1&2\end{pmatrix}=A^{-1}$ exactly, and on the 200-sample sine + sawtooth it converged in 4 iterations with $|\text{corr}|=1.000$ to each source.

**Strengths**

- Sensors record linear mixtures of physically independent processes: EEG/MEG, fMRI, audio with several microphones, spectroscopy.
- You need to remove one specific artifact (eye blinks, line noise, heartbeat) while keeping the rest of the signal.
- PCA gives components that are uncorrelated but not interpretable, and you suspect non-Gaussian underlying factors.
- You want sparse, localized features (ICA on natural image patches gives edge-like Gabor filters).

**Limitations**

- The sources are (close to) Gaussian — the problem is unidentifiable and components are arbitrary rotations.
- Mixing is not instantaneous: echoes, delays and reverberation (convolutive mixtures) need frequency-domain ICA or other methods.
- You need the actual amplitude, sign or ranking of components — ICA cannot give them.
- There are fewer sensors than sources (underdetermined), or the mixing changes over time.
- Your goal is just compression or visualization: PCA, UMAP or t-SNE are simpler and better suited.

- **EEG / MEG artifact removal:** Decompose multichannel recordings, identify the components with eye-blink or muscle topographies, zero them, and remix the clean signal.
- **fMRI brain networks:** Spatial ICA on resting-state scans separates independent networks (default mode, visual, motor) without a task model.
- **Audio source separation:** Separate speakers or instruments recorded by several microphones; in the frequency domain for real rooms.
- **Finance and telecom:** Extract independent factors driving many asset returns; separate users' signals in multi-antenna receivers.

> **Sign, scale and order are arbitrary:** Component 1 in one run can be component 3, negated, in the next. Never interpret the sign or magnitude of a component by itself — look at the component together with its column of the mixing matrix (their product is invariant), and match components across runs by absolute correlation.

> **Gaussian-ish data gives meaningless components:** FastICA will always return *something*. If the outputs have near-zero kurtosis, the algorithm warns about convergence, or reruns with different seeds give different components, the sources are not identifiable. Check the non-Gaussianity of each component and its stability across seeds before interpreting it.

> **Too many components:** Asking for as many components as channels on noisy data forces ICA to split noise into spurious, unstable components (overlearning). Reduce with `n_components` (the whitening step keeps the top PCA directions), and make sure you have many more samples than squared dimensions.

> **Scaling matters less than you think:** Unlike PCA, ICA is unaffected by rescaling individual sensors: any invertible scaling is absorbed into A, and whitening normalizes the variance anyway. Centring is required; standardization is optional.

**Question:** Why does FastICA whiten the data before searching for components?

- A. Whitening makes the sources Gaussian, which ICA requires
- B. After whitening the remaining unmixing is an orthogonal matrix (a rotation), so far fewer parameters remain to be found
- C. Whitening removes the sign ambiguity
- D. Whitening is the same as ICA when the data is two-dimensional

**Answer:** B. After whitening the remaining unmixing is an orthogonal matrix (a rotation), so far fewer parameters remain to be found — With $z=Vx$ white and unit-variance sources, $(VA)(VA)^\top=I$, so only an orthogonal matrix — $d(d-1)/2$ angles, one angle in 2-D — is left. The sign ambiguity remains, and whitening alone (PCA) gave kurtosis −1.2 instead of −2 in the worked example.

**Question:** Two independent ±1 sources (kurtosis −2 each) are combined as y = 0.6 s₁ + 0.8 s₂. What is the kurtosis of y?

- A. −2
- B. −1.0784
- C. −1.4
- D. 0

**Answer:** B. −1.0784 — $\kappa(y)=\sum_i q_i^4\kappa(s_i)=-2(0.6^4+0.8^4)=-2(0.1296+0.4096)=-1.0784$. The mixture is closer to Gaussian (0) than either source — the central-limit effect ICA exploits.

**Question:** Both sources are Gaussian. What does ICA recover?

- A. Both sources, up to sign
- B. One source; the other stays mixed
- C. Nothing identifiable: every rotation of the whitened data is equally independent
- D. The PCA components, which are then the sources

**Answer:** C. Nothing identifiable: every rotation of the whitened data is equally independent — A rotated pair of independent unit Gaussians is again a pair of independent unit Gaussians, so no objective can prefer one rotation. In the figure the |kurtosis| curve stays below 0.22 and FastICA never converges. ICA tolerates at most one Gaussian source.

**Question:** Two sources with equal kurtosis; the current w is off from a source by an angle whose tangent is 0.5 (26.57°). After one cube-FastICA update the error is about…

- A. 13.3°
- B. 7.13°
- C. 1.0°
- D. 26.57° (no change)

**Answer:** B. 7.13° — In source coordinates $q_i^+\propto\kappa_i q_i^3$, so with equal kurtoses the tangent of the error is cubed: $0.5^3=0.125$, and $\arctan 0.125=7.13°$. That cubic convergence is why the worked example needed only three updates.

**Key points**

- ICA models $x=As$ with independent, non-Gaussian sources and learns $W\approx A^{-1}$; sources come back only up to order, sign and scale, and at most one may be Gaussian.
- Whitening (PCA + rescaling) decorrelates the data and leaves only a rotation; PCA stops there and returns mixtures.
- By the central limit theorem mixtures are more Gaussian than sources, so ICA rotates to maximize non-Gaussianity: kurtosis $\sum_i q_i^4\kappa_i$ or a logcosh negentropy approximation.
- FastICA is an approximate Newton fixed point $w^+=E[zg(w^\top z)]-E[g'(w^\top z)]w$ plus normalization/decorrelation — no learning rate, cubic or quadratic convergence.
- Check stability: near-Gaussian components, convergence warnings or seed-dependent results mean the components are not meaningful.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
