NMF

Factor a non-negative matrix as X ≈ WH with W, H ≥ 0, giving additive, parts-based components.

17 min read1510 wordscore5 viewsupdated 2026-10-07

Definition

Non-negative matrix factorization (NMF) approximates a data matrix with no negative entries, X∈R≥0n×dX\in\mathbb{R}_{\ge0}^{n\times d}, by the product of two smaller non-negative matrices, X≈WHX\approx WH with W∈R≥0n×kW\in\mathbb{R}_{\ge0}^{n\times k} and H∈R≥0k×dH\in\mathbb{R}_{\ge0}^{k\times d}. Each row of HH is a part (a topic, a face feature, a spectral template), and each row of WW says how much of each part a sample contains. Because nothing can be subtracted, every sample is rebuilt purely by adding parts.

At a glance

Task
Unsupervised dimensionality reduction / matrix factorization; soft clustering and topic extraction
Input → output
non-negative XX (nn samples × dd features) → WW (n×kn\times k weights) and HH (k×dk\times d parts)
Learns
kk non-negative basis vectors (rows of HH) and a non-negative code per sample (rows of WW)
Objective
12∥X−WH∥F2\tfrac12\lVert X-WH\rVert_F^2 (default), or KL / Itakura–Saito divergence; non-convex, solved by alternating updates
Training
O(n d k)O(n\,d\,k) per iteration (sparse XX: O(nnz⋅k)O(\text{nnz}\cdot k)); tens to hundreds of iterations
Prediction
new row xx: a small non-negative least squares problem with HH fixed, O(d k)O(d\,k) per inner step
Hyperparameters
rank k, loss (Frobenius / KL / IS), initialization (random, NNDSVD), solver (mu, cd), L1/L2 penalties, iterations
Assumes
all entries ≥ 0 and meaningful as amounts (counts, intensities, power); data is a sum of a few non-negative sources

The problem

Many data sets are amounts: word counts in documents, pixel brightness in images, power in an audio spectrogram, gene expression levels, ratings. Such data is naturally a sum of parts — a document about a football club's charity bake-off is some sports vocabulary plus some cooking vocabulary.

PCA and the SVD find the best low-rank approximation, but their components mix signs. On the six-document example below, PCA's first component is (0.48, 0.40, 0.25, −0.47, −0.42, −0.39)(0.48,\,0.40,\,0.25,\,-0.47,\,-0.42,\,-0.39) over goal, team, match, oven, flour, recipe: a contrast "sports minus cooking". A document is then rebuilt by adding some components and subtracting others, so a component on its own means little — and a reconstruction can even contain negative word counts. We want components that look like the things in the data, combined only by adding.

The idea

Forbid negative numbers in both factors. Then Xij≈∑tWitHtjX_{ij}\approx\sum_t W_{it}H_{tj} is a sum of non-negative terms: no cancellation is possible, so a part can only be used where its words (pixels, frequencies) actually occur. To explain a count of zero, every part used by that sample must be (near) zero on that feature. The cheapest way to satisfy this is to make parts that cover disjoint groups of features and samples that use only a few parts — so NMF tends to produce sparse, localized, interpretable components without being asked to. Lee and Seung (1999) showed it on faces: PCA gives ghostly whole "eigenfaces", NMF gives noses, eyes and mouths.

Vocabulary

Rank k
number of parts (topics, components); the inner dimension of WHWH
Basis / dictionary H
k×dk\times d; row tt is part tt expressed over the original features (a topic's word weights)
Coefficients / activations W
n×kn\times k; row ii says how much of each part sample ii uses (a document's topic mix)
Reconstruction
X^=WH\hat X=WH; the residual is X−WHX-WH
Parts-based
each sample is an additive combination of parts; parts are not cancelled by other parts
Multiplicative update
an update that multiplies each entry by a positive ratio, so a positive entry stays positive
Fold-in
computing WW for new samples with HH held fixed (scikit-learn's transform)
NNDSVD
non-negative double SVD — a deterministic initialization built from the positive parts of the leading singular vectors

The math

min⁡W≥0,  H≥0  L(W,H)=12∥X−WH∥F2=12∑i=1n∑j=1d(Xij−∑t=1kWitHtj)2\min_{W\ge0,\;H\ge0}\;\mathcal{L}(W,H)=\tfrac12\lVert X-WH\rVert_F^2=\tfrac12\sum_{i=1}^{n}\sum_{j=1}^{d}\Big(X_{ij}-\sum_{t=1}^{k}W_{it}H_{tj}\Big)^2
XX
n×dn\times d data matrix, all entries ≥0\ge 0
WW
n×kn\times k coefficient matrix, Wit≥0W_{it}\ge0
HH
k×dk\times d basis matrix, Htj≥0H_{tj}\ge0
kk
rank — number of parts, usually k≪min⁡(n,d)k\ll\min(n,d)
∥⋅∥F\lVert\cdot\rVert_F
Frobenius norm: square root of the sum of squared entries

The problem is non-convex in (W,H)(W,H) jointly (and NP-hard in general), but it is convex in HH for fixed WW and in WW for fixed HH — each half is a non-negative least squares problem. So every practical solver alternates: improve HH with WW fixed, then WW with HH fixed.

Step 1 — the gradient. Differentiating the loss with respect to HH (and symmetrically WW):

∇HL=W⊤WH−W⊤X,∇WL=WHH⊤−XH⊤\nabla_H\mathcal{L}=W^\top W H-W^\top X,\qquad \nabla_W\mathcal{L}=W H H^\top-X H^\top
W⊤XW^\top X
positive part of the gradient's negative: how strongly the data "pulls" each HtjH_{tj} up
W⊤WHW^\top W H
how much of feature jj the current reconstruction already supplies through part tt

Step 2 — a gradient step with a clever step size. Ordinary gradient descent H←H−η ∇HLH\leftarrow H-\eta\,\nabla_H\mathcal{L} can push entries below zero. Lee and Seung chose a different step size for every entry, ηtj=Htj/(W⊤WH)tj\eta_{tj}=H_{tj}/(W^\top WH)_{tj}:

Htj←Htj−Htj(W⊤WH)tj[(W⊤WH)tj−(W⊤X)tj]=Htj (W⊤X)tj(W⊤WH)tjH_{tj}\leftarrow H_{tj}-\frac{H_{tj}}{(W^\top WH)_{tj}}\Big[(W^\top WH)_{tj}-(W^\top X)_{tj}\Big]=H_{tj}\,\frac{(W^\top X)_{tj}}{(W^\top WH)_{tj}}
ηtj\eta_{tj}
entry-wise step size; it shrinks to 0 as the entry approaches 0, so the step can never overshoot below zero

The HtjH_{tj} terms cancel and the subtraction disappears, leaving a multiplicative rule. In matrix form, with ⊙\odot and ⊘\oslash for element-wise product and division:

H←H⊙W⊤XW⊤WH,W←W⊙XH⊤WHH⊤H\leftarrow H\odot\frac{W^\top X}{W^\top W H},\qquad W\leftarrow W\odot\frac{X H^\top}{W H H^\top}
⊙, ⊘\odot,\ \oslash
element-wise multiplication and division
ratio>1\text{ratio}>1
the data wants more of this entry → it grows
ratio<1\text{ratio}<1
the reconstruction overshoots → it shrinks
ratio=1\text{ratio}=1
gradient is zero for that entry (a KKT point when Htj>0H_{tj}>0)

Every quantity in the ratio is non-negative, so non-negativity is preserved for free — no projection needed. Lee and Seung proved, with an auxiliary-function (majorize–minimize) argument, that each update never increases the loss. It does not guarantee the global minimum: different initializations can land in different local minima.

Other losses. For count data the Poisson-motivated (generalized) Kullback–Leibler divergence is often a better fit, and gives an equally simple update:

DKL(X ∥ WH)=∑ij(Xijlog⁡Xij(WH)ij−Xij+(WH)ij),H←H⊙W⊤(X⊘WH)W⊤1D_{KL}(X\,\|\,WH)=\sum_{ij}\Big(X_{ij}\log\frac{X_{ij}}{(WH)_{ij}}-X_{ij}+(WH)_{ij}\Big),\qquad H\leftarrow H\odot\frac{W^\top\big(X\oslash WH\big)}{W^\top\mathbf 1}
DKLD_{KL}
generalized KL divergence; equals the negative Poisson log-likelihood up to constants
1\mathbf 1
n×dn\times d matrix of ones

Both are members of the β-divergence family: β=2\beta=2 is Frobenius, β=1\beta=1 is KL, β=0\beta=0 is Itakura–Saito (scale-invariant, used for audio spectrograms). Smaller β\beta pays more attention to small entries.

Lreg=12∥X−WH∥F2+α ρ (∥W∥1+∥H∥1)+α(1−ρ)2(∥W∥F2+∥H∥F2)\mathcal{L}_{\text{reg}}=\tfrac12\lVert X-WH\rVert_F^2+\alpha\,\rho\,\big(\lVert W\rVert_1+\lVert H\rVert_1\big)+\tfrac{\alpha(1-\rho)}{2}\big(\lVert W\rVert_F^2+\lVert H\rVert_F^2\big)
α\alpha
overall regularization strength
ρ\rho
L1 ratio (scikit-learn's l1_ratio): 1 = pure L1 (sparsity), 0 = pure L2 (small, smooth entries)

With penalties the denominators simply gain the penalty's gradient: L1 adds αρ\alpha\rho, L2 adds α(1−ρ)H\alpha(1-\rho)H. A larger denominator shrinks every entry each step, and entries with weak support are pushed to zero — the L1 term is the main knob for sparser parts.

Worked example: topics in six documents

Six short documents, counted over six words. d1–d2 are about football, d3–d4 about baking, d5–d6 mix both. We ask for k=2k=2 parts and run the multiplicative updates from a fixed positive start.

Docgoalteammatchovenflourrecipe
d1322000
d2231001
d3000322
d4010233
d5221121
d6102212
X: word counts. Highlighted: the two mixed documents.
DocW₀ topic 1W₀ topic 2·H₀ rowgoalteammatchovenflourrecipe
d10.60.4topic 10.90.70.80.40.30.5
d20.80.3topic 20.30.40.20.80.90.6
d30.30.7
d40.40.9
d50.50.5
d60.70.6
The starting point. Its product is a blurry guess with loss 12∥X−W0H0∥F2=25.71\tfrac12\lVert X-W_0H_0\rVert_F^2=25.71.

Iteration 1 by hand (W first, then H — the order scikit-learn uses)

  1. 1Numerator for W. XH0⊤XH_0^\top, row d1: goal·0.9 + team·0.7 + match·0.8 =3(0.9)+2(0.7)+2(0.8)=5.70=3(0.9)+2(0.7)+2(0.8)=5.70 for topic 1, and 3(0.3)+2(0.4)+2(0.2)=2.103(0.3)+2(0.4)+2(0.2)=2.10 for topic 2.
  2. 2Denominator for W. W0H0H0⊤W_0H_0H_0^\top, row d1: (2.104,  1.800)(2.104,\;1.800) — what the current reconstruction already "explains" through each topic.
  3. 3Update row d1. W11=0.6×5.70/2.104=1.6255W_{11}=0.6\times5.70/2.104=1.6255 (the data wants much more topic 1) and W12=0.4×2.10/1.800=0.4667W_{12}=0.4\times2.10/1.800=0.4667 (ratio 1.171.17, a small increase). All six rows: d1 (1.63,0.47)(1.63,0.47), d2 (1.71,0.41)(1.71,0.41), d3 (0.45,1.94)(0.45,1.94), d4 (0.65,2.31)(0.65,2.31), d5 (1.36,1.30)(1.36,1.30), d6 (1.21,1.11)(1.21,1.11) — the sports and baking documents already separate.
  4. 4Update H with the new W. Column goal: W⊤X=(12.2271,  5.9206)W^\top X=(12.2271,\;5.9206), W⊤WH0=(10.6309,  9.9614)W^\top WH_0=(10.6309,\;9.9614). So H1,goal=0.9×12.2271/10.6309=1.0351H_{1,\text{goal}}=0.9\times12.2271/10.6309=1.0351 and H2,goal=0.3×5.9206/9.9614=0.1783H_{2,\text{goal}}=0.3\times5.9206/9.9614=0.1783: goal moves into topic 1 and fades from topic 2.
  5. 5Loss after one iteration: 25.71→9.8325.71\to9.83. Repeating: 6.096.09 (iteration 2), 4.714.71 (3), 4.154.15 (5), 4.024.02 (10), 3.993.99 (20) and 3.98213.9821 from iteration 50 on — relative error ∥X−WH∥F/∥X∥F=28.1%\lVert X-WH\rVert_F/\lVert X\rVert_F=28.1\%.
DocW topic 1W topic 2share (sports / baking)
d12.190100% / 0%
d21.960.2390% / 10%
d302.280% / 100%
d40.252.609% / 91%
d51.561.2256% / 44%
d60.751.6232% / 68%
After 500 iterations. Entries shown as 0 are below 10−3010^{-30} — driven to zero by the updates. Normalizing each row of W gives the topic share of each document.
Topicgoalteammatchovenflourrecipe
topic 1 (sports)1.231.140.7700.030.12
topic 2 (baking)00.060.091.021.011.02
The learned parts H: one sports topic and one baking topic, with almost no overlap.

Fold in a new document: goal ×2, oven ×1, recipe ×1

  1. 1Keep HH fixed and solve min⁡h≥0∥x−hH∥2\min_{h\ge0}\lVert x-hH\rVert^2 for x=(2,0,0,1,0,1)x=(2,0,0,1,0,1) — a 2-variable non-negative least squares problem.
  2. 2Normal equations: HH⊤=(3.42650.29120.29123.1295)HH^\top=\begin{pmatrix}3.4265&0.2912\\0.2912&3.1295\end{pmatrix}, Hx=(2.5784,  2.0483)Hx=(2.5784,\;2.0483).
  3. 3Solve: h=(HH⊤)−1Hx=(0.7024,  0.5891)h=(HH^\top)^{-1}Hx=(0.7024,\;0.5891). Both entries are positive, so the constraint is inactive and this is the NNLS solution.
  4. 4Interpretation: 54% sports, 46% baking. The reconstruction hH=(0.86,0.84,0.59,0.60,0.62,0.69)hH=(0.86,0.84,0.59,0.60,0.62,0.69) spreads each topic's mass over all its words, with residual norm 1.731.73 — NMF summarizes, it does not memorize.
number of topics k:
▸ NMF multiplicative updates · k = 2step 1/14
X (6 docs × 6 words) ≈ W · H · topics: ● 1 ● 2 goalgoalteamteammatchmatchovenovenflourflourreciperecipeH (topic → word)0.900.700.800.400.300.500.300.400.200.800.900.60d10.600.40d20.800.30d30.300.70d40.400.90d50.500.50d60.700.600.70.60.60.60.50.50.80.70.70.60.50.60.50.50.40.70.70.60.60.60.50.90.90.70.60.60.50.60.60.60.80.70.70.80.80.7W H (reconstruction)W≈322000231001000322010233221121102212X (word counts)loss ½‖X − WH‖² vs iteration (log scale) · - - best rank-2 (SVD) = 3.91011010050001326
iteration 0loss 25.71relative error 71.3%SVD bound 3.91
  1. 1W, H ← small positive values
  2. 2repeat:
  3. 3 W ← W ⊙ (X Hᵀ) ⊘ (W H Hᵀ)
  4. 4 H ← H ⊙ (Wᵀ X) ⊘ (Wᵀ W H)
  5. 5until ½‖X − WH‖² stops falling
  6. 6new x: h = argmin_{h ≥ 0} ‖x − h H‖

Start from strictly positive W (6×2) and H (2×6). Their product is a blurry guess: loss ½‖X − WH‖² = 25.71.

fig.Multiplicative updates on the six-document matrix. W (documents × topics) sits left of the product, H (topics × words) above it, and the reconstruction WH sits where matrix multiplication puts it, next to the data X; each WH cell is coloured by the mix of topics that build it. Step through the two half-steps of iterations 1–2 (W₁₁ = 0.60 × 5.70 / 2.10 = 1.63, then H₁₁ = 0.90 × 12.23 / 10.63 = 1.04) and then iterations 3 … 500: the loss falls 25.71 → 9.83 → 3.98 against the dashed SVD bound 3.91 while unused entries decay to 0. The last step folds in the new document, h = (0.70, 0.59). Switch k to 1 (loss 18.69, equal to the bound) or 3 (1.78 vs 1.22).

Watch the colours: after a few iterations the WH cells for d1–d2 are almost pure topic-1 blue and d3–d4 almost pure topic-2 orange, while d5–d6 are visibly blended — that blend is the additive combination NMF is built on. With k=3k=3 the sports topic splits in two (one led by goal and match, one by team), which is the first sign of choosing kk too large for six documents.

The algorithm, step by step

NMF(X, k) with multiplicative updates

  1. 1Check the input. Every entry of XX must be ≥0\ge0. Optionally scale features (e.g. TF-IDF for text), but never centre — centring creates negatives.
  2. 2Initialize WW (n×kn\times k) and HH (k×dk\times d) with strictly positive values: random uniform scaled to Xˉ/k\sqrt{\bar X/k}, or NNDSVD (init="nndsvda" fills its zeros with the data mean, which multiplicative updates need).
  3. 3Update W with HH fixed: W←W⊙(XH⊤)⊘(WHH⊤+ε)W\leftarrow W\odot (XH^\top)\oslash(WHH^\top+\varepsilon), where a tiny ε\varepsilon guards against division by zero.
  4. 4Update H with WW fixed: H←H⊙(W⊤X)⊘(W⊤WH+ε)H\leftarrow H\odot (W^\top X)\oslash(W^\top WH+\varepsilon).
  5. 5Check convergence. Stop when the relative drop in loss over the last few iterations is below a tolerance (scikit-learn default tol=1e-4) or at max_iter.
  6. 6Normalize (optional): scale each row of HH to unit sum or norm and absorb the scale into WW.
  7. 7Transform new data: with HH fixed, solve min⁡h≥0∥x−hH∥\min_{h\ge0}\lVert x-hH\rVert per row (repeat the WW-update alone, or use NNLS).

Cost

Per iteration
Compute XH⊤XH^\top and W⊤XW^\top X in O(ndk)O(ndk); the small Gram matrices HH⊤HH^\top, W⊤WW^\top W in O((n+d)k2)O((n+d)k^2). Multiply W(HH⊤)W(HH^\top) rather than (WH)H⊤(WH)H^\top to avoid forming the dense n×dn\times d product.
Sparse X
O(nnz(X) k)O(\text{nnz}(X)\,k) per iteration — NMF scales to millions of documents with sparse TF-IDF matrices.
Memory
O((n+d)k)O((n+d)k) for the factors plus XX itself.
Iterations
multiplicative updates: often hundreds; coordinate descent (HALS): typically tens (14 on the worked example, to the same loss 3.9821).

Variants and tuning

KnobOptionsEffect
Rank k2 … a few hundredMore parts → lower error but finer, less distinct parts. On the example: loss 18.69 (k=1), 3.98 (k=2), 1.78 (k=3).
Loss (beta_loss)frobenius · kullback-leibler · itakura-saitoFrobenius for intensities / general data; KL for counts (Poisson); IS for audio power spectra. KL and IS need the mu solver.
Solvercd (HALS, coordinate descent) · mucd converges much faster for Frobenius; mu is the only option for other β-divergences.
Initializationnndsvd · nndsvda · nndsvdar · randomNNDSVD is deterministic and starts near a good solution; random needs several restarts. Use nndsvda/nndsvdar with mu (no exact zeros).
L1 penalty (alpha_W, alpha_H)0 … largeSparser parts and codes. Error ‖X−WH‖ 2.82 → 2.92 → 7.56 and zeros in H 2 → 3 → 9 of 12 for α = 0, 0.1, 0.5.
L2 penaltyl1_ratio < 1Smaller, smoother entries; stabilizes ill-conditioned problems.
The choices that matter, roughly in order of impact. Penalty numbers: scikit-learn, worked example, NNDSVDa start, l1_ratio=1.

Choosing k

Elbow
plot reconstruction error against k and look for where the gains flatten.
Stability
run several seeds per k and measure how consistent the parts are (consensus clustering, as used in genomics); stable k values are trustworthy.
Held-out entries
mask a random 5–10% of entries, fit on the rest (weighted NMF), and pick the k with the lowest error on the masked ones.
Downstream
if the parts feed a classifier or a recommender, choose k by that task's validation score; for topics, by topic coherence.

Related models

Sparse NMF
explicit L1 (or Hoyer sparseness) constraints on W or H.
Semi-NMF / convex NMF
allow XX (and HH) to have mixed signs while keeping W≥0W\ge0, or restrict parts to convex combinations of data points.
Probabilistic view
KL-NMF is maximum likelihood for Poisson counts; with normalized factors it is equivalent to PLSA, the precursor of LDA topic models.
Online / mini-batch NMF
scikit-learn's MiniBatchNMF updates H from batches, for data that does not fit in memory.
Tensor / convolutive NMF
non-negative CP decomposition for 3-way data; convolutive NMF for patterns that extend over time.

In code

python
1import numpy as np
2
3def nmf(X, k, iters=500, W=None, H=None, seed=0, eps=1e-12):
4 """Lee–Seung multiplicative updates for min ½‖X − WH‖², W, H ≥ 0."""
5 rng = np.random.default_rng(seed)
6 n, d = X.shape
7 W = rng.uniform(0.1, 1, (n, k)) if W is None else W.astype(float).copy()
8 H = rng.uniform(0.1, 1, (k, d)) if H is None else H.astype(float).copy()
9 losses = [0.5 * np.sum((X - W @ H) ** 2)]
10 for _ in range(iters):
11 W *= (X @ H.T) / (W @ H @ H.T + eps) # gradient step with step size W / (W H Hᵀ)
12 H *= (W.T @ X) / (W.T @ W @ H + eps) # same for H
13 losses.append(0.5 * np.sum((X - W @ H) ** 2))
14 return W, H, losses
15
16def fold_in(x, H, iters=2000, eps=1e-12):
17 """Topic weights of a new row with H fixed: the same update applied to h alone."""
18 h = np.ones(H.shape[0])
19 for _ in range(iters):
20 h *= (H @ x) / (H @ H.T @ h + eps)
21 return h
22
23words = ["goal", "team", "match", "oven", "flour", "recipe"]
24X = np.array([[3, 2, 2, 0, 0, 0],
25 [2, 3, 1, 0, 0, 1],
26 [0, 0, 0, 3, 2, 2],
27 [0, 1, 0, 2, 3, 3],
28 [2, 2, 1, 1, 2, 1],
29 [1, 0, 2, 2, 1, 2]], dtype=float)
30W0 = np.array([[.6, .4], [.8, .3], [.3, .7], [.4, .9], [.5, .5], [.7, .6]])
31H0 = np.array([[.9, .7, .8, .4, .3, .5], [.3, .4, .2, .8, .9, .6]])
32
33W, H, losses = nmf(X, 2, W=W0, H=H0)
34print("loss:", [round(float(losses[i]), 2) for i in (0, 1, 2, 3, 500)]) # 25.71 9.83 6.09 4.71 3.98
35print("H =\n", H.round(2))
36for t in range(2):
37 print(f"topic {t + 1}:", [words[j] for j in np.argsort(-H[t])[:3]])
38print("d5 topic share:", (W[4] / W[4].sum()).round(2)) # [0.56 0.44]
39print("new doc h =", fold_in(np.array([2, 0, 0, 1, 0, 1.]), H).round(4)) # [0.7024 0.5891]
From scratch in NumPy: Lee–Seung updates on the worked example, plus fold-in of a new document with the same update applied to h alone.
python
1import numpy as np
2from sklearn.decomposition import NMF
3
4X = np.array([[3, 2, 2, 0, 0, 0],
5 [2, 3, 1, 0, 0, 1],
6 [0, 0, 0, 3, 2, 2],
7 [0, 1, 0, 2, 3, 3],
8 [2, 2, 1, 1, 2, 1],
9 [1, 0, 2, 2, 1, 2]], dtype=float)
10W0 = np.array([[.6, .4], [.8, .3], [.3, .7], [.4, .9], [.5, .5], [.7, .6]])
11H0 = np.array([[.9, .7, .8, .4, .3, .5], [.3, .4, .2, .8, .9, .6]])
12
13# 1) Exactly the article's run: multiplicative updates from the same start.
14mu = NMF(n_components=2, init="custom", solver="mu", max_iter=500, tol=0)
15W = mu.fit_transform(X, W=W0.copy(), H=H0.copy())
16print("loss =", round(0.5 * mu.reconstruction_err_ ** 2, 4)) # 3.9821
17print("H =\n", mu.components_.round(2))
18print("new doc h =", mu.transform([[2, 0, 0, 1, 0, 1]]).round(4)) # [[0.7024 0.5891]]
19
20# 2) What you would use in practice: NNDSVD start + coordinate descent.
21cd = NMF(n_components=2, init="nndsvda", solver="cd", max_iter=1000, random_state=0).fit(X)
22print("cd: loss =", round(0.5 * cd.reconstruction_err_ ** 2, 4), "in", cd.n_iter_, "iterations")
scikit-learn: the identical run (init='custom', solver='mu') reproduces every number, and the practical default (NNDSVDa + coordinate descent) reaches the same loss in 14 iterations.

Use when

  • +Your data are non-negative amounts — counts, intensities, spectra, expression levels — and you want components a person can read.
  • +You need soft, overlapping memberships: a document can be 56% one topic and 44% another.
  • +The matrix is large and sparse (TF-IDF, user–item); NMF works directly on sparse input.
  • +You want a compact, non-negative feature set for a downstream model or for source separation.

Avoid when

  • −The data have meaningful negative values or must be centred — use PCA, ICA or Factor Analysis.
  • −You need a unique, reproducible answer: results depend on initialization and k, up to scaling and permutation.
  • −You need the optimal low-rank reconstruction — truncated SVD is always at least as accurate.
  • −The goal is a 2-D picture of neighbourhoods — use t-SNE or UMAP.

Where it's used

Topic modelling

Factor a TF-IDF document–term matrix: rows of H are topics (ranked word lists), rows of W are document–topic mixtures. A fast, deterministic alternative to LDA.

Audio source separation

Factor a magnitude spectrogram (frequency × time): H holds spectral templates of notes or instruments, W their activations over time; mask and invert to separate sources.

Genomics

Gene-expression and mutation-count matrices factor into metagenes and mutational signatures (COSMIC signatures were extracted with NMF).

Recommenders and images

Non-negative user–item factors give interpretable taste profiles; image patches and faces decompose into localized parts; hyperspectral pixels unmix into material spectra.

Check yourself

In a multiplicative update, an entry has Htj=0.5H_{tj}=0.5, (W⊤X)tj=6(W^\top X)_{tj}=6 and (W⊤WH)tj=4(W^\top WH)_{tj}=4. What is its new value?

Check yourself

You initialize one entry of H to exactly 0. After 1,000 multiplicative updates it is…

Check yourself

(W,H)(W,H) minimizes the NMF loss. Which pair has exactly the same loss?

Check yourself

On the worked example, rank-2 NMF reaches loss 3.98 while the truncated SVD reaches 3.91. Why can NMF never beat the SVD at the same rank?

Remember

  • ✓NMF factors a non-negative matrix as X≈WHX\approx WH with W,H≥0W,H\ge0: rows of HH are parts, rows of WW are how much of each part a sample contains.
  • ✓No subtraction means no cancellation, so parts come out additive, localized and often sparse — readable topics, face parts, spectral templates.
  • ✓Lee–Seung updates H←H⊙W⊤X⊘W⊤WHH\leftarrow H\odot W^\top X\oslash W^\top WH (and the mirror for WW) are gradient steps with entry-wise step sizes; they keep entries non-negative and never increase the loss.
  • ✓The problem is non-convex and the factors are unique only up to scaling and permutation: initialize with NNDSVD or several seeds, and normalize before comparing.
  • ✓Choose the loss to match the data (Frobenius, KL for counts, Itakura–Saito for audio), choose k by error elbow, stability or a downstream task, and never centre the data.
← back to ML Algorithms roadmap