# Matrix Multiplication

> A matrix is a linear map; multiplying two matrices composes the two maps.

Source: https://www.mlfoundry.in/learn/matrix-multiplication  
Section: Mathematics › Linear Algebra  
Level: core · 6 min read · updated 2026-10-03

**Matrix multiplication:** combines an $m\times n$ matrix $A$ and an $n\times p$ matrix $B$ into the $m\times p$ matrix $C=AB$, whose entry $c_{ij}$ is the dot product of row $i$ of $A$ with column $j$ of $B$. As maps, $AB$ is the single linear map that applies $B$ first and then $A$.

- **Notation:** $C=AB$, $\ c_{ij}=\sum_k a_{ik}b_{kj}$
- **Input → output:** $(m\times n)\cdot(n\times p)\to m\times p$ — inner dimensions must match
- **Cost:** $mnp$ multiply-adds; $O(n^3)$ for square matrices
- **Key properties:** associative and distributive; **not** commutative; $(AB)^\top=B^\top A^\top$; $\det(AB)=\det A\det B$
- **Prerequisites:** vectors & the dot product
- **Used in:** every neural-network layer, batch prediction, PCA, Markov chains, graphics

## The intuition

A matrix is a machine that moves vectors: $x\mapsto Ax$. It is completely described by where it sends the basis vectors — **column $j$ of $A$ is the image of $e_j$**. Everything else follows by linearity, so lines stay straight, the origin stays put, and grid lines stay parallel and evenly spaced, though the grid may be stretched, sheared, rotated or flipped.

Two moves in a row are again a linear move, and its matrix is the product. Order matters for the same reason that "rotate, then shear" and "shear, then rotate" end in different places.

## The math

$$
c_{ij}=\sum_{k=1}^{n}a_{ik}\,b_{kj},\qquad A\in\mathbb{R}^{m\times n},\ B\in\mathbb{R}^{n\times p},\ C\in\mathbb{R}^{m\times p}
$$

- $a_{ik}$: entry in row $i$, column $k$ of $A$
- $n$: the shared inner dimension

**Three views of the same product**

1. **Dot products:** $c_{ij}$ = (row $i$ of $A$) · (column $j$ of $B$).
2. **Columns:** $Ax=x_1a_1+\dots+x_na_n$ is a weighted sum of $A$'s columns — so column $j$ of $AB$ is $A$ applied to column $j$ of $B$.
3. **Composition:** $(AB)x=A(Bx)$ for every $x$. The definition is chosen precisely to make this true.

The **determinant** is the factor by which a square matrix scales area (volume in higher dimensions); its sign says whether orientation flips. For a $2\times2$ matrix:

$$
\det\begin{pmatrix}a&b\\c&d\end{pmatrix}=ad-bc,\qquad \det(AB)=\det A\cdot\det B
$$

Composition multiplies scale factors. A determinant of zero means the map flattens the plane onto a line or a point: information is lost, and no matrix can undo it.

## See it

> Figure: Drag the tips of î and ĵ — they are the columns of A. The shaded square's area is det A; it turns rose when orientation flips and grey at det = 0 (try 'project'). The compose buttons play shear-then-rotate and rotate-then-shear: different results.

## Worked example

Shear $A=\begin{pmatrix}1&1\\0&1\end{pmatrix}$, rotation by 90° $B=\begin{pmatrix}0&-1\\1&0\end{pmatrix}$. Then $AB=\begin{pmatrix}1&-1\\1&0\end{pmatrix}$ but $BA=\begin{pmatrix}0&-1\\1&1\end{pmatrix}$. Rotate-then-shear sends $(1,0)$ to $(1,1)$; shear-then-rotate sends it to $(0,1)$. Both products still have determinant $1\cdot1=1$: neither map changes area.

## In code

```python
import numpy as np

A = np.array([[1, 1], [0, 1]])        # shear
B = np.array([[0, -1], [1, 0]])       # rotate 90°
A @ B                                 # [[1, -1], [1, 0]]  rotate, then shear
B @ A                                 # [[0, -1], [1, 1]]  shear, then rotate
np.linalg.det(A @ B)                  # ≈ 1.0 — area preserved

# dense layer on a batch: (batch, in) @ (in, out) -> (batch, out)
X = np.random.randn(32, 784)
W = np.random.randn(784, 128) * 0.01
H = X @ W                             # (32, 128): 32·784·128 ≈ 3.2M multiply-adds
```

Order matters; a dense layer is one matrix product over a batch.

- **Neural networks:** A dense layer computes $XW$; a forward pass is a chain of matrix products with nonlinearities between them.
- **Batch prediction:** One product scores every example against every weight vector at once — the workload GPUs are built for.
- **Dimensionality reduction:** PCA maps data to $k$ dimensions with one product, $XV_k$.
- **Graphs & Markov chains:** $P^t$ gives $t$-step transition probabilities; entries of $A^k$ count length-$k$ paths in a graph.

> **AB ≠ BA:** In general the two products differ, and often only one is defined. Read products right to left: $ABx$ applies $B$ first.

> *** is not @:** In NumPy, `A * B` multiplies entry by entry (the Hadamard product); `A @ B` is the matrix product. When the shapes happen to broadcast, the wrong one runs silently.

> **Parenthesization changes cost:** $(AB)x$ and $A(Bx)$ are equal, but for $1000\times1000$ matrices the first costs about $10^9$ multiply-adds and the second $2\times10^6$. Multiply by the vector first.

**Question:** $A$ is $3\times4$ and $B$ is $4\times2$. What is the shape of $AB$, and is $BA$ defined?

- A. 3×2; BA is undefined
- B. 4×4; BA is 2×3
- C. 3×2; BA is 2×4
- D. 2×3; BA is undefined

**Answer:** A. 3×2; BA is undefined — Inner dimensions $4=4$ match, giving $3\times2$. For $BA$ the inner dimensions are $2$ and $3$, which don't match.

**Question:** $\det A=2$ and $\det B=-3$. What is $\det(AB)$, and what does $B$ do to orientation?

- A. −6; B flips orientation
- B. 6; B preserves orientation
- C. −1; B flips orientation
- D. −6; B preserves orientation

**Answer:** A. −6; B flips orientation — $\det(AB)=\det A\det B=-6$. A negative determinant means orientation is reversed.

**Key points**

- $c_{ij}$ = row $i$ of $A$ · column $j$ of $B$; inner dimensions must match.
- Columns of $A$ are the images of the basis vectors; $Ax$ mixes the columns.
- $AB$ means apply $B$, then $A$ — not commutative.
- $\det$ = area scale factor; $\det(AB)=\det A\det B$; zero means information lost.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
