# Linear Systems, Inverse & Rank

> Ax = b has a unique solution exactly when A is square with full rank; rank counts the independent directions A can reach.

Source: https://www.mlfoundry.in/learn/linear-systems-inverse-rank  
Section: Mathematics › Linear Algebra  
Level: core · 6 min read · updated 2026-10-03

**A linear system:** $Ax=b$ asks for the weights $x$ that combine the columns of $A$ into $b$. The **rank** of $A$ is its number of linearly independent columns (equivalently, rows) — the dimension of everything $A$ can reach. A square matrix of full rank has an **inverse** $A^{-1}$ with $A^{-1}A=I$, and then $x=A^{-1}b$ is the only solution, for every $b$.

- **Notation:** $A\in\mathbb{R}^{m\times n}$, $x\in\mathbb{R}^n$, $b\in\mathbb{R}^m$; $\operatorname{rank}A$; $A^{-1}$
- **Solvable iff:** $b$ lies in the column space of $A$
- **Unique iff:** also $\operatorname{rank}A=n$ — no nonzero $x$ with $Ax=0$
- **Invertible iff:** square and full rank ⇔ $\det A\ne0$
- **Cost:** Gaussian elimination (LU), $O(n^3)$
- **Prerequisites:** matrix multiplication
- **Used in:** least squares, Newton's method, Gaussian processes, Kalman filters

## The intuition

The same equations have two pictures. **Row picture:** each equation is a line (a hyperplane in higher dimensions), and a solution is a point lying on all of them. **Column picture:** $Ax=x_1a_1+x_2a_2$, so solving means finding how much of each column to mix to land exactly on $b$.

Trouble starts when columns are redundant. If $a_2$ is a multiple of $a_1$, mixing them reaches only one line. A $b$ off that line is unreachable — no solution. A $b$ on it is reachable in infinitely many ways, by trading some $a_1$ for some $a_2$. Rank counts how many genuinely different directions the columns provide.

## The math

The **column space** $C(A)$ is every vector of the form $Ax$; the **null space** $N(A)$ is every $x$ with $Ax=0$. The rank–nullity theorem links them:

$$
\operatorname{rank}A+\dim N(A)=n
$$

- $n$: number of columns (unknowns)
- $N(A)$: directions $x$ that $A$ sends to $0$

So the solution set is either empty ($b\notin C(A)$) or one particular solution plus the whole null space, $x=x_p+N(A)$. It is a single point exactly when $N(A)=\{0\}$, i.e. $\operatorname{rank}A=n$. A tall matrix can satisfy this but reaches only some $b$; a square full-rank matrix reaches every $b$. For square matrices, full rank, $\det A\ne0$ and invertibility are the same condition.

$$
\begin{pmatrix}a&b\\c&d\end{pmatrix}^{-1}=\frac{1}{ad-bc}\begin{pmatrix}d&-b\\-c&a\end{pmatrix}
$$

- $ad-bc$: $\det A$ — the inverse exists only if it is nonzero

The $1/\det A$ factor hints at the real danger: **near**-singular matrices. The **condition number** $\kappa(A)=\sigma_{\max}/\sigma_{\min}$, the ratio of the largest to the smallest singular value, bounds how much relative error in $b$ is amplified in $x$. Float64 carries about 16 digits, so $\kappa=10^{10}$ leaves roughly 6 trustworthy digits.

**Gaussian elimination**

1. Subtract multiples of the first row to zero the first column below the pivot.
2. Repeat on the remaining rows and columns until the matrix is upper-triangular.
3. Back-substitute from the last equation upward.
4. The number of nonzero pivots is the rank.

## See it

> Figure: Left: each equation is a line; the solution is where they cross. Right: b = x₁a₁ + x₂a₂ drawn head to tail. Starting at a₁ = (2, 1), a₂ = (1, 3), b = (3, 5): det = 5 and x = (0.8, 1.4). Make the columns parallel and rank drops to 1 — the lines become parallel (no solution) or identical (infinitely many).

## Worked example

$2x_1+x_2=3$ and $x_1+3x_2=5$. Here $\det A=2\cdot3-1\cdot1=5$, $A^{-1}=\tfrac15\begin{pmatrix}3&-1\\-1&2\end{pmatrix}$, so $x=\tfrac15(9-5,\,-3+10)=(0.8,\,1.4)$. Check: $2(0.8)+1.4=3$ and $0.8+3(1.4)=5$.

Now $x_1+2x_2=3$ and $2x_1+4x_2=6$: the second row is twice the first, so $\det A=0$ and rank is 1. Every point $x=(3,0)+t(-2,1)$ solves it. Change the right side to $(3,7)$ and the rows contradict each other ($2\cdot3\ne7$): no solution.

## In code

```python
import numpy as np

A = np.array([[2.0, 1.0], [1.0, 3.0]]); b = np.array([3.0, 5.0])
np.linalg.solve(A, b)                       # array([0.8, 1.4]) — LU, no inverse formed
np.linalg.matrix_rank(A), np.linalg.cond(A) # (2, ≈ 2.618)

S = np.array([[1.0, 2.0], [2.0, 4.0]])      # parallel columns
np.linalg.matrix_rank(S)                    # 1
np.linalg.lstsq(S, np.array([3.0, 6.0]), rcond=None)[0]   # ≈ [0.6, 1.2] (min-norm)
```

Solve, inspect rank and conditioning, and handle the singular case.

- **Least squares:** The normal equations $X^\top Xw=X^\top y$ are a linear system; collinear features make $X$ rank-deficient and $w$ non-unique.
- **Second-order optimization:** Each Newton step solves $H\,\Delta=-\nabla f$ for the step $\Delta$.
- **Gaussian processes & Kalman filters:** Predictions require solving with covariance matrices, usually via a Cholesky factorization.
- **Feature engineering:** One-hot encoding every category plus an intercept is rank-deficient — the dummy-variable trap.

> **Never invert to solve:** `np.linalg.inv(A) @ b` is slower and less accurate than `np.linalg.solve(A, b)`, which factorizes A once and back-substitutes.

> **det is not a singularity test:** $\det(0.1\,I_{100})=10^{-100}$, yet that matrix has $\kappa=1$ — perfectly conditioned. Judge near-singularity by the condition number or the smallest singular values.

**Question:** $A$ is $3\times3$ with rank 2, and $b$ is in its column space. How many solutions does $Ax=b$ have?

- A. exactly one
- B. none
- C. infinitely many
- D. exactly two

**Answer:** C. infinitely many — Rank–nullity: $\dim N(A)=3-2=1$, so solutions form a line $x_p+tn$.

**Question:** What is $\det\begin{pmatrix}1&2\\3&4\end{pmatrix}$, and is the matrix invertible?

- A. −2, invertible
- B. 2, invertible
- C. −2, not invertible
- D. 10, invertible

**Answer:** A. −2, invertible — $1\cdot4-2\cdot3=-2\ne0$, so the matrix is invertible.

**Key points**

- Solvable iff $b\in C(A)$; unique iff also $\operatorname{rank}A=n$.
- Rank = independent columns; $\operatorname{rank}A+\dim N(A)=n$.
- Square: full rank ⇔ $\det A\ne0$ ⇔ invertible.
- Solve with `solve` or `lstsq`, never `inv`; judge stability by the condition number.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
