# NMF

> Factor a non-negative matrix as X ≈ WH with W, H ≥ 0, giving additive, parts-based components.

Source: https://www.mlfoundry.in/learn/nmf  
Section: ML Algorithms › Unsupervised · Dimensionality Reduction  
Level: core · 17 min read · updated 2026-10-07

**Non-negative matrix factorization (NMF):** approximates a data matrix with no negative entries, $X\in\mathbb{R}_{\ge0}^{n\times d}$, by the product of two smaller non-negative matrices, $X\approx WH$ with $W\in\mathbb{R}_{\ge0}^{n\times k}$ and $H\in\mathbb{R}_{\ge0}^{k\times d}$. Each row of $H$ is a **part** (a topic, a face feature, a spectral template), and each row of $W$ says **how much of each part** a sample contains. Because nothing can be subtracted, every sample is rebuilt purely by *adding* parts.

- **Task:** Unsupervised dimensionality reduction / matrix factorization; soft clustering and topic extraction
- **Input → output:** non-negative $X$ ($n$ samples × $d$ features) → $W$ ($n\times k$ weights) and $H$ ($k\times d$ parts)
- **Learns:** $k$ non-negative basis vectors (rows of $H$) and a non-negative code per sample (rows of $W$)
- **Objective:** $\tfrac12\lVert X-WH\rVert_F^2$ (default), or KL / Itakura–Saito divergence; non-convex, solved by alternating updates
- **Training:** $O(n\,d\,k)$ per iteration (sparse $X$: $O(\text{nnz}\cdot k)$); tens to hundreds of iterations
- **Prediction:** new row $x$: a small non-negative least squares problem with $H$ fixed, $O(d\,k)$ per inner step
- **Hyperparameters:** rank k, loss (Frobenius / KL / IS), initialization (random, NNDSVD), solver (mu, cd), L1/L2 penalties, iterations
- **Assumes:** all entries ≥ 0 and meaningful as amounts (counts, intensities, power); data is a sum of a few non-negative sources

## The problem

Many data sets are *amounts*: word counts in documents, pixel brightness in images, power in an audio spectrogram, gene expression levels, ratings. Such data is naturally a **sum of parts** — a document about a football club's charity bake-off is some sports vocabulary *plus* some cooking vocabulary.

PCA and the SVD find the best low-rank approximation, but their components mix signs. On the six-document example below, PCA's first component is $(0.48,\,0.40,\,0.25,\,-0.47,\,-0.42,\,-0.39)$ over *goal, team, match, oven, flour, recipe*: a **contrast** "sports minus cooking". A document is then rebuilt by adding some components and *subtracting* others, so a component on its own means little — and a reconstruction can even contain negative word counts. We want components that look like the things in the data, combined only by adding.

## The idea

Forbid negative numbers in both factors. Then $X_{ij}\approx\sum_t W_{it}H_{tj}$ is a sum of non-negative terms: **no cancellation is possible**, so a part can only be used where its words (pixels, frequencies) actually occur. To explain a count of zero, every part used by that sample must be (near) zero on that feature. The cheapest way to satisfy this is to make parts that cover *disjoint groups* of features and samples that use only a few parts — so NMF tends to produce **sparse, localized, interpretable** components without being asked to. Lee and Seung (1999) showed it on faces: PCA gives ghostly whole "eigenfaces", NMF gives noses, eyes and mouths.

**Vocabulary**

- **Rank k:** number of parts (topics, components); the inner dimension of $WH$
- **Basis / dictionary H:** $k\times d$; row $t$ is part $t$ expressed over the original features (a topic's word weights)
- **Coefficients / activations W:** $n\times k$; row $i$ says how much of each part sample $i$ uses (a document's topic mix)
- **Reconstruction:** $\hat X=WH$; the residual is $X-WH$
- **Parts-based:** each sample is an additive combination of parts; parts are not cancelled by other parts
- **Multiplicative update:** an update that *multiplies* each entry by a positive ratio, so a positive entry stays positive
- **Fold-in:** computing $W$ for new samples with $H$ held fixed (scikit-learn's `transform`)
- **NNDSVD:** non-negative double SVD — a deterministic initialization built from the positive parts of the leading singular vectors

> **How it relates to its neighbours:** PCA / truncated SVD give the *best* rank-$k$ fit with orthogonal, mixed-sign components. **ICA** keeps mixed signs but asks the sources to be statistically independent; **Factor Analysis** adds a Gaussian noise model per feature. NMF's only extra assumption is non-negativity. **t-SNE** and **UMAP** are for visualizing neighbourhoods, not for decomposing a matrix into parts — see those articles for that goal.

## The math

$$
\min_{W\ge0,\;H\ge0}\;\mathcal{L}(W,H)=\tfrac12\lVert X-WH\rVert_F^2=\tfrac12\sum_{i=1}^{n}\sum_{j=1}^{d}\Big(X_{ij}-\sum_{t=1}^{k}W_{it}H_{tj}\Big)^2
$$

- $X$: $n\times d$ data matrix, all entries $\ge 0$
- $W$: $n\times k$ coefficient matrix, $W_{it}\ge0$
- $H$: $k\times d$ basis matrix, $H_{tj}\ge0$
- $k$: rank — number of parts, usually $k\ll\min(n,d)$
- $\lVert\cdot\rVert_F$: Frobenius norm: square root of the sum of squared entries

The problem is **non-convex** in $(W,H)$ jointly (and NP-hard in general), but it is convex in $H$ for fixed $W$ and in $W$ for fixed $H$ — each half is a non-negative least squares problem. So every practical solver **alternates**: improve $H$ with $W$ fixed, then $W$ with $H$ fixed.

**Step 1 — the gradient.** Differentiating the loss with respect to $H$ (and symmetrically $W$):

$$
\nabla_H\mathcal{L}=W^\top W H-W^\top X,\qquad \nabla_W\mathcal{L}=W H H^\top-X H^\top
$$

- $W^\top X$: positive part of the gradient's negative: how strongly the data "pulls" each $H_{tj}$ up
- $W^\top W H$: how much of feature $j$ the current reconstruction already supplies through part $t$

**Step 2 — a gradient step with a clever step size.** Ordinary gradient descent $H\leftarrow H-\eta\,\nabla_H\mathcal{L}$ can push entries below zero. Lee and Seung chose a *different step size for every entry*, $\eta_{tj}=H_{tj}/(W^\top WH)_{tj}$:

$$
H_{tj}\leftarrow H_{tj}-\frac{H_{tj}}{(W^\top WH)_{tj}}\Big[(W^\top WH)_{tj}-(W^\top X)_{tj}\Big]=H_{tj}\,\frac{(W^\top X)_{tj}}{(W^\top WH)_{tj}}
$$

- $\eta_{tj}$: entry-wise step size; it shrinks to 0 as the entry approaches 0, so the step can never overshoot below zero

The $H_{tj}$ terms cancel and the subtraction disappears, leaving a **multiplicative** rule. In matrix form, with $\odot$ and $\oslash$ for element-wise product and division:

$$
H\leftarrow H\odot\frac{W^\top X}{W^\top W H},\qquad W\leftarrow W\odot\frac{X H^\top}{W H H^\top}
$$

- $\odot,\ \oslash$: element-wise multiplication and division
- $\text{ratio}>1$: the data wants more of this entry → it grows
- $\text{ratio}<1$: the reconstruction overshoots → it shrinks
- $\text{ratio}=1$: gradient is zero for that entry (a KKT point when $H_{tj}>0$)

Every quantity in the ratio is non-negative, so non-negativity is preserved for free — no projection needed. Lee and Seung proved, with an auxiliary-function (majorize–minimize) argument, that **each update never increases the loss**. It does *not* guarantee the global minimum: different initializations can land in different local minima.

> **Why zeros appear on their own:** A multiplicative update can shrink an entry geometrically towards 0 but can never revive an *exact* zero — a zero times any ratio stays zero. Entries the data does not support decay towards 0 over the iterations, which is how NMF ends up with sparse factors. (The flip side: never initialize with exact zeros, or those entries are frozen forever.)

**Other losses.** For count data the Poisson-motivated (generalized) Kullback–Leibler divergence is often a better fit, and gives an equally simple update:

$$
D_{KL}(X\,\|\,WH)=\sum_{ij}\Big(X_{ij}\log\frac{X_{ij}}{(WH)_{ij}}-X_{ij}+(WH)_{ij}\Big),\qquad H\leftarrow H\odot\frac{W^\top\big(X\oslash WH\big)}{W^\top\mathbf 1}
$$

- $D_{KL}$: generalized KL divergence; equals the negative Poisson log-likelihood up to constants
- $\mathbf 1$: $n\times d$ matrix of ones

Both are members of the **β-divergence** family: $\beta=2$ is Frobenius, $\beta=1$ is KL, $\beta=0$ is Itakura–Saito (scale-invariant, used for audio spectrograms). Smaller $\beta$ pays more attention to small entries.

$$
\mathcal{L}_{\text{reg}}=\tfrac12\lVert X-WH\rVert_F^2+\alpha\,\rho\,\big(\lVert W\rVert_1+\lVert H\rVert_1\big)+\tfrac{\alpha(1-\rho)}{2}\big(\lVert W\rVert_F^2+\lVert H\rVert_F^2\big)
$$

- $\alpha$: overall regularization strength
- $\rho$: L1 ratio (scikit-learn's l1_ratio): 1 = pure L1 (sparsity), 0 = pure L2 (small, smooth entries)

With penalties the denominators simply gain the penalty's gradient: L1 adds $\alpha\rho$, L2 adds $\alpha(1-\rho)H$. A larger denominator shrinks every entry each step, and entries with weak support are pushed to zero — the L1 term is the main knob for **sparser parts**.

> **The factors are not unique:** For any positive diagonal $D$ (or a permutation), $(WD)(D^{-1}H)=WH$: you can rescale part $t$ by 2 and halve its coefficients, or swap two parts, without changing the fit. Fix a convention before comparing runs — e.g. normalize each row of $H$ to sum to 1 and move the scale into $W$ — and match components by similarity, not by index.

## Worked example: topics in six documents

Six short documents, counted over six words. d1–d2 are about football, d3–d4 about baking, d5–d6 mix both. We ask for $k=2$ parts and run the multiplicative updates from a fixed positive start.

X: word counts. Highlighted: the two mixed documents.

| Doc | goal | team | match | oven | flour | recipe |
| --- | --- | --- | --- | --- | --- | --- |
| d1 | 3 | 2 | 2 | 0 | 0 | 0 |
| d2 | 2 | 3 | 1 | 0 | 0 | 1 |
| d3 | 0 | 0 | 0 | 3 | 2 | 2 |
| d4 | 0 | 1 | 0 | 2 | 3 | 3 |
| d5 | 2 | 2 | 1 | 1 | 2 | 1 |
| d6 | 1 | 0 | 2 | 2 | 1 | 2 |

The starting point. Its product is a blurry guess with loss $\tfrac12\lVert X-W_0H_0\rVert_F^2=25.71$.

| Doc | W₀ topic 1 | W₀ topic 2 | · | H₀ row | goal | team | match | oven | flour | recipe |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| d1 | 0.6 | 0.4 |  | topic 1 | 0.9 | 0.7 | 0.8 | 0.4 | 0.3 | 0.5 |
| d2 | 0.8 | 0.3 |  | topic 2 | 0.3 | 0.4 | 0.2 | 0.8 | 0.9 | 0.6 |
| d3 | 0.3 | 0.7 |  |  |  |  |  |  |  |  |
| d4 | 0.4 | 0.9 |  |  |  |  |  |  |  |  |
| d5 | 0.5 | 0.5 |  |  |  |  |  |  |  |  |
| d6 | 0.7 | 0.6 |  |  |  |  |  |  |  |  |

**Iteration 1 by hand (W first, then H — the order scikit-learn uses)**

1. **Numerator for W.** $XH_0^\top$, row d1: goal·0.9 + team·0.7 + match·0.8 $=3(0.9)+2(0.7)+2(0.8)=5.70$ for topic 1, and $3(0.3)+2(0.4)+2(0.2)=2.10$ for topic 2.
2. **Denominator for W.** $W_0H_0H_0^\top$, row d1: $(2.104,\;1.800)$ — what the current reconstruction already "explains" through each topic.
3. **Update row d1.** $W_{11}=0.6\times5.70/2.104=1.6255$ (the data wants much more topic 1) and $W_{12}=0.4\times2.10/1.800=0.4667$ (ratio $1.17$, a small increase). All six rows: d1 $(1.63,0.47)$, d2 $(1.71,0.41)$, d3 $(0.45,1.94)$, d4 $(0.65,2.31)$, d5 $(1.36,1.30)$, d6 $(1.21,1.11)$ — the sports and baking documents already separate.
4. **Update H with the new W.** Column *goal*: $W^\top X=(12.2271,\;5.9206)$, $W^\top WH_0=(10.6309,\;9.9614)$. So $H_{1,\text{goal}}=0.9\times12.2271/10.6309=1.0351$ and $H_{2,\text{goal}}=0.3\times5.9206/9.9614=0.1783$: *goal* moves into topic 1 and fades from topic 2.
5. **Loss after one iteration:** $25.71\to9.83$. Repeating: $6.09$ (iteration 2), $4.71$ (3), $4.15$ (5), $4.02$ (10), $3.99$ (20) and $3.9821$ from iteration 50 on — relative error $\lVert X-WH\rVert_F/\lVert X\rVert_F=28.1\%$.

After 500 iterations. Entries shown as 0 are below $10^{-30}$ — driven to zero by the updates. Normalizing each row of W gives the topic share of each document.

| Doc | W topic 1 | W topic 2 | share (sports / baking) |
| --- | --- | --- | --- |
| d1 | 2.19 | 0 | 100% / 0% |
| d2 | 1.96 | 0.23 | 90% / 10% |
| d3 | 0 | 2.28 | 0% / 100% |
| d4 | 0.25 | 2.60 | 9% / 91% |
| d5 | 1.56 | 1.22 | 56% / 44% |
| d6 | 0.75 | 1.62 | 32% / 68% |

The learned parts H: one sports topic and one baking topic, with almost no overlap.

| Topic | goal | team | match | oven | flour | recipe |
| --- | --- | --- | --- | --- | --- | --- |
| topic 1 (sports) | **1.23** | **1.14** | **0.77** | 0 | 0.03 | 0.12 |
| topic 2 (baking) | 0 | 0.06 | 0.09 | **1.02** | **1.01** | **1.02** |

**Fold in a new document: goal ×2, oven ×1, recipe ×1**

1. Keep $H$ fixed and solve $\min_{h\ge0}\lVert x-hH\rVert^2$ for $x=(2,0,0,1,0,1)$ — a 2-variable non-negative least squares problem.
2. Normal equations: $HH^\top=\begin{pmatrix}3.4265&0.2912\\0.2912&3.1295\end{pmatrix}$, $Hx=(2.5784,\;2.0483)$.
3. Solve: $h=(HH^\top)^{-1}Hx=(0.7024,\;0.5891)$. Both entries are positive, so the constraint is inactive and this *is* the NNLS solution.
4. Interpretation: 54% sports, 46% baking. The reconstruction $hH=(0.86,0.84,0.59,0.60,0.62,0.69)$ spreads each topic's mass over all its words, with residual norm $1.73$ — NMF summarizes, it does not memorize.

> **How close to optimal?:** The best rank-2 approximation with *no* sign constraint (truncated SVD, by Eckart–Young) has loss $3.9121$; NMF reaches $3.9821$ — only 1.8% worse, but with parts you can read. With $k=3$ the gap widens ($1.78$ vs $1.22$), and with $k=1$ there is none at all ($18.69$ both): for a non-negative matrix the top singular vectors can be chosen non-negative (Perron–Frobenius), so rank-1 NMF *is* rank-1 SVD.

> Figure: Multiplicative updates on the six-document matrix. W (documents × topics) sits left of the product, H (topics × words) above it, and the reconstruction WH sits where matrix multiplication puts it, next to the data X; each WH cell is coloured by the mix of topics that build it. Step through the two half-steps of iterations 1–2 (W₁₁ = 0.60 × 5.70 / 2.10 = 1.63, then H₁₁ = 0.90 × 12.23 / 10.63 = 1.04) and then iterations 3 … 500: the loss falls 25.71 → 9.83 → 3.98 against the dashed SVD bound 3.91 while unused entries decay to 0. The last step folds in the new document, h = (0.70, 0.59). Switch k to 1 (loss 18.69, equal to the bound) or 3 (1.78 vs 1.22).

Watch the colours: after a few iterations the WH cells for d1–d2 are almost pure topic-1 blue and d3–d4 almost pure topic-2 orange, while d5–d6 are visibly blended — that blend is the *additive* combination NMF is built on. With $k=3$ the sports topic splits in two (one led by *goal* and *match*, one by *team*), which is the first sign of choosing $k$ too large for six documents.

## The algorithm, step by step

**NMF(X, k) with multiplicative updates**

1. **Check the input.** Every entry of $X$ must be $\ge0$. Optionally scale features (e.g. TF-IDF for text), but never centre — centring creates negatives.
2. **Initialize** $W$ ($n\times k$) and $H$ ($k\times d$) with strictly positive values: random uniform scaled to $\sqrt{\bar X/k}$, or NNDSVD (`init="nndsvda"` fills its zeros with the data mean, which multiplicative updates need).
3. **Update W** with $H$ fixed: $W\leftarrow W\odot (XH^\top)\oslash(WHH^\top+\varepsilon)$, where a tiny $\varepsilon$ guards against division by zero.
4. **Update H** with $W$ fixed: $H\leftarrow H\odot (W^\top X)\oslash(W^\top WH+\varepsilon)$.
5. **Check convergence.** Stop when the relative drop in loss over the last few iterations is below a tolerance (scikit-learn default `tol=1e-4`) or at `max_iter`.
6. **Normalize** (optional): scale each row of $H$ to unit sum or norm and absorb the scale into $W$.
7. **Transform new data:** with $H$ fixed, solve $\min_{h\ge0}\lVert x-hH\rVert$ per row (repeat the $W$-update alone, or use NNLS).

**Cost**

- **Per iteration:** Compute $XH^\top$ and $W^\top X$ in $O(ndk)$; the small Gram matrices $HH^\top$, $W^\top W$ in $O((n+d)k^2)$. Multiply $W(HH^\top)$ rather than $(WH)H^\top$ to avoid forming the dense $n\times d$ product.
- **Sparse X:** $O(\text{nnz}(X)\,k)$ per iteration — NMF scales to millions of documents with sparse TF-IDF matrices.
- **Memory:** $O((n+d)k)$ for the factors plus $X$ itself.
- **Iterations:** multiplicative updates: often hundreds; coordinate descent (HALS): typically tens (14 on the worked example, to the same loss 3.9821).

## Variants and tuning

The choices that matter, roughly in order of impact. Penalty numbers: scikit-learn, worked example, NNDSVDa start, `l1_ratio=1`.

| Knob | Options | Effect |
| --- | --- | --- |
| Rank k | 2 … a few hundred | More parts → lower error but finer, less distinct parts. On the example: loss 18.69 (k=1), 3.98 (k=2), 1.78 (k=3). |
| Loss (beta_loss) | frobenius · kullback-leibler · itakura-saito | Frobenius for intensities / general data; KL for counts (Poisson); IS for audio power spectra. KL and IS need the mu solver. |
| Solver | cd (HALS, coordinate descent) · mu | cd converges much faster for Frobenius; mu is the only option for other β-divergences. |
| Initialization | nndsvd · nndsvda · nndsvdar · random | NNDSVD is deterministic and starts near a good solution; random needs several restarts. Use nndsvda/nndsvdar with mu (no exact zeros). |
| L1 penalty (alpha_W, alpha_H) | 0 … large | Sparser parts and codes. Error ‖X−WH‖ 2.82 → 2.92 → 7.56 and zeros in H 2 → 3 → 9 of 12 for α = 0, 0.1, 0.5. |
| L2 penalty | l1_ratio < 1 | Smaller, smoother entries; stabilizes ill-conditioned problems. |

**Choosing k**

- **Elbow:** plot reconstruction error against k and look for where the gains flatten.
- **Stability:** run several seeds per k and measure how consistent the parts are (consensus clustering, as used in genomics); stable k values are trustworthy.
- **Held-out entries:** mask a random 5–10% of entries, fit on the rest (weighted NMF), and pick the k with the lowest error on the masked ones.
- **Downstream:** if the parts feed a classifier or a recommender, choose k by that task's validation score; for topics, by topic coherence.

**Related models**

- **Sparse NMF:** explicit L1 (or Hoyer sparseness) constraints on W or H.
- **Semi-NMF / convex NMF:** allow $X$ (and $H$) to have mixed signs while keeping $W\ge0$, or restrict parts to convex combinations of data points.
- **Probabilistic view:** KL-NMF is maximum likelihood for Poisson counts; with normalized factors it is equivalent to PLSA, the precursor of LDA topic models.
- **Online / mini-batch NMF:** scikit-learn's MiniBatchNMF updates H from batches, for data that does not fit in memory.
- **Tensor / convolutive NMF:** non-negative CP decomposition for 3-way data; convolutive NMF for patterns that extend over time.

## In code

```python
import numpy as np

def nmf(X, k, iters=500, W=None, H=None, seed=0, eps=1e-12):
    """Lee–Seung multiplicative updates for min ½‖X − WH‖²,  W, H ≥ 0."""
    rng = np.random.default_rng(seed)
    n, d = X.shape
    W = rng.uniform(0.1, 1, (n, k)) if W is None else W.astype(float).copy()
    H = rng.uniform(0.1, 1, (k, d)) if H is None else H.astype(float).copy()
    losses = [0.5 * np.sum((X - W @ H) ** 2)]
    for _ in range(iters):
        W *= (X @ H.T) / (W @ H @ H.T + eps)        # gradient step with step size W / (W H Hᵀ)
        H *= (W.T @ X) / (W.T @ W @ H + eps)        # same for H
        losses.append(0.5 * np.sum((X - W @ H) ** 2))
    return W, H, losses

def fold_in(x, H, iters=2000, eps=1e-12):
    """Topic weights of a new row with H fixed: the same update applied to h alone."""
    h = np.ones(H.shape[0])
    for _ in range(iters):
        h *= (H @ x) / (H @ H.T @ h + eps)
    return h

words = ["goal", "team", "match", "oven", "flour", "recipe"]
X = np.array([[3, 2, 2, 0, 0, 0],
              [2, 3, 1, 0, 0, 1],
              [0, 0, 0, 3, 2, 2],
              [0, 1, 0, 2, 3, 3],
              [2, 2, 1, 1, 2, 1],
              [1, 0, 2, 2, 1, 2]], dtype=float)
W0 = np.array([[.6, .4], [.8, .3], [.3, .7], [.4, .9], [.5, .5], [.7, .6]])
H0 = np.array([[.9, .7, .8, .4, .3, .5], [.3, .4, .2, .8, .9, .6]])

W, H, losses = nmf(X, 2, W=W0, H=H0)
print("loss:", [round(float(losses[i]), 2) for i in (0, 1, 2, 3, 500)])   # 25.71 9.83 6.09 4.71 3.98
print("H =\n", H.round(2))
for t in range(2):
    print(f"topic {t + 1}:", [words[j] for j in np.argsort(-H[t])[:3]])
print("d5 topic share:", (W[4] / W[4].sum()).round(2))           # [0.56 0.44]
print("new doc h =", fold_in(np.array([2, 0, 0, 1, 0, 1.]), H).round(4))  # [0.7024 0.5891]
```

From scratch in NumPy: Lee–Seung updates on the worked example, plus fold-in of a new document with the same update applied to h alone.

```python
import numpy as np
from sklearn.decomposition import NMF

X = np.array([[3, 2, 2, 0, 0, 0],
              [2, 3, 1, 0, 0, 1],
              [0, 0, 0, 3, 2, 2],
              [0, 1, 0, 2, 3, 3],
              [2, 2, 1, 1, 2, 1],
              [1, 0, 2, 2, 1, 2]], dtype=float)
W0 = np.array([[.6, .4], [.8, .3], [.3, .7], [.4, .9], [.5, .5], [.7, .6]])
H0 = np.array([[.9, .7, .8, .4, .3, .5], [.3, .4, .2, .8, .9, .6]])

# 1) Exactly the article's run: multiplicative updates from the same start.
mu = NMF(n_components=2, init="custom", solver="mu", max_iter=500, tol=0)
W = mu.fit_transform(X, W=W0.copy(), H=H0.copy())
print("loss =", round(0.5 * mu.reconstruction_err_ ** 2, 4))   # 3.9821
print("H =\n", mu.components_.round(2))
print("new doc h =", mu.transform([[2, 0, 0, 1, 0, 1]]).round(4))  # [[0.7024 0.5891]]

# 2) What you would use in practice: NNDSVD start + coordinate descent.
cd = NMF(n_components=2, init="nndsvda", solver="cd", max_iter=1000, random_state=0).fit(X)
print("cd: loss =", round(0.5 * cd.reconstruction_err_ ** 2, 4), "in", cd.n_iter_, "iterations")
```

scikit-learn: the identical run (init='custom', solver='mu') reproduces every number, and the practical default (NNDSVDa + coordinate descent) reaches the same loss in 14 iterations.

> **reconstruction_err_ is not the loss:** scikit-learn reports `reconstruction_err_` as the Frobenius norm ‖X − WH‖ (2.8221 here), not ½‖X − WH‖². Square and halve it to compare with the loss in this article (3.9821).

**Strengths**

- Your data are non-negative amounts — counts, intensities, spectra, expression levels — and you want components a person can read.
- You need soft, overlapping memberships: a document can be 56% one topic and 44% another.
- The matrix is large and sparse (TF-IDF, user–item); NMF works directly on sparse input.
- You want a compact, non-negative feature set for a downstream model or for source separation.

**Limitations**

- The data have meaningful negative values or must be centred — use PCA, ICA or Factor Analysis.
- You need a unique, reproducible answer: results depend on initialization and k, up to scaling and permutation.
- You need the optimal low-rank reconstruction — truncated SVD is always at least as accurate.
- The goal is a 2-D picture of neighbourhoods — use t-SNE or UMAP.

- **Topic modelling:** Factor a TF-IDF document–term matrix: rows of H are topics (ranked word lists), rows of W are document–topic mixtures. A fast, deterministic alternative to LDA.
- **Audio source separation:** Factor a magnitude spectrogram (frequency × time): H holds spectral templates of notes or instruments, W their activations over time; mask and invert to separate sources.
- **Genomics:** Gene-expression and mutation-count matrices factor into metagenes and mutational signatures (COSMIC signatures were extracted with NMF).
- **Recommenders and images:** Non-negative user–item factors give interpretable taste profiles; image patches and faces decompose into localized parts; hyperspectral pixels unmix into material spectra.

> **Do not centre or standardize first:** Subtracting the mean (as you would before PCA) creates negative entries and NMF will reject the input. Scale instead: TF-IDF or length-normalize documents, divide each feature by its maximum, or log1p counts. The data must stay $\ge0$.

> **One run is not an answer:** The objective is non-convex. With random initialization, two runs can give different parts, and a bad run can sit in a poor local minimum. Use NNDSVD for determinism, or run several seeds and keep the lowest loss — and check that the parts you interpret are stable across runs.

**Question:** In a multiplicative update, an entry has $H_{tj}=0.5$, $(W^\top X)_{tj}=6$ and $(W^\top WH)_{tj}=4$. What is its new value?

- A. 0.5 − 6 + 4 = −1.5, clipped to 0
- B. 0.75
- C. 0.33
- D. 2.0

**Answer:** B. 0.75 — $H_{tj}\leftarrow 0.5\times6/4=0.75$. The ratio is above 1 (the data asks for more than the reconstruction supplies), so the entry grows — and a multiplicative update can never go negative.

**Question:** You initialize one entry of H to exactly 0. After 1,000 multiplicative updates it is…

- A. Still exactly 0
- B. Positive if the data supports it
- C. Negative
- D. Equal to the SVD value

**Answer:** A. Still exactly 0 — Zero times any ratio is zero, so the entry is frozen forever. This is why NNDSVD initializations used with the mu solver fill their zeros with small positive values (nndsvda / nndsvdar).

**Question:** $(W,H)$ minimizes the NMF loss. Which pair has exactly the same loss?

- A. $(2W,\;2H)$
- B. $(W+1,\;H)$
- C. $(2W,\;\tfrac12H)$
- D. $(W^\top,\;H^\top)$

**Answer:** C. $(2W,\;\tfrac12H)$ — $(2W)(\tfrac12H)=WH$. Any positive diagonal rescaling $(WD)(D^{-1}H)$, and any reordering of the parts, leaves the product — and the loss — unchanged, so NMF factors are only defined up to scale and permutation.

**Question:** On the worked example, rank-2 NMF reaches loss 3.98 while the truncated SVD reaches 3.91. Why can NMF never beat the SVD at the same rank?

- A. NMF uses fewer iterations
- B. The SVD is the best rank-k approximation over all real matrices; NMF searches only the non-negative ones
- C. The SVD also uses non-negative factors
- D. NMF minimizes a different norm

**Answer:** B. The SVD is the best rank-k approximation over all real matrices; NMF searches only the non-negative ones — By the Eckart–Young theorem the truncated SVD minimizes $\lVert X-\hat X\rVert_F$ over every rank-$k$ $\hat X$. NMF's product $WH$ is one such matrix, so its error is at least as large; the non-negativity is the price paid for interpretability.

**Key points**

- NMF factors a non-negative matrix as $X\approx WH$ with $W,H\ge0$: rows of $H$ are parts, rows of $W$ are how much of each part a sample contains.
- No subtraction means no cancellation, so parts come out additive, localized and often sparse — readable topics, face parts, spectral templates.
- Lee–Seung updates $H\leftarrow H\odot W^\top X\oslash W^\top WH$ (and the mirror for $W$) are gradient steps with entry-wise step sizes; they keep entries non-negative and never increase the loss.
- The problem is non-convex and the factors are unique only up to scaling and permutation: initialize with NNDSVD or several seeds, and normalize before comparing.
- Choose the loss to match the data (Frobenius, KL for counts, Itakura–Saito for audio), choose k by error elbow, stability or a downstream task, and never centre the data.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
