# Vectors & the Dot Product

> a·b = ‖a‖‖b‖cos θ — the dot product measures alignment, and it is the core operation of every neural layer.

Source: https://www.mlfoundry.in/learn/vectors-dot-product  
Section: Mathematics › Linear Algebra  
Level: core · 5 min read · updated 2026-10-03

**The dot product:** of two vectors $a, b\in\mathbb{R}^n$ is the number $a\cdot b=\sum_{i=1}^n a_ib_i$ — multiply matching coordinates and add. Geometrically it equals $\lVert a\rVert\,\lVert b\rVert\cos\theta$, where $\theta$ is the angle between them.

- **Notation:** $a\cdot b$, $a^\top b$, $\langle a, b\rangle$
- **Input → output:** two vectors of the same length → one number
- **Cost:** $n$ multiplications, $n-1$ additions
- **Key property:** $a\cdot b=0$ ⇔ orthogonal (for nonzero vectors); $a\cdot a=\lVert a\rVert^2$
- **Prerequisites:** coordinates, Pythagoras, cosine
- **Used in:** neurons, attention, cosine similarity, projections, every matrix product

## The intuition

A vector is an arrow from the origin — or, equally, a list of $n$ numbers such as one row of a dataset. The dot product asks **how much two arrows point the same way, weighted by their lengths**. Same direction gives a large positive number, perpendicular gives exactly zero, opposite gives a large negative number.

Read as arithmetic, $w\cdot x$ is a weighted sum: each feature $x_i$ times its importance $w_i$. That single operation is a linear model, one neuron, and one entry of every matrix product.

## The math

$$
a\cdot b=\sum_{i=1}^{n}a_ib_i,\qquad \lVert a\rVert=\sqrt{a\cdot a}
$$

- $a_i$: $i$-th coordinate of $a$
- $\lVert a\rVert$: Euclidean length of a

Directly from the sum: it is symmetric ($a\cdot b=b\cdot a$), linear in each argument ($(\alpha a+c)\cdot b=\alpha\,a\cdot b+c\cdot b$), and $a\cdot a\ge0$ with equality only for $a=0$.

**Why the cosine.** The triangle with sides $a$, $b$ and $a-b$ obeys the law of cosines, $\lVert a-b\rVert^2=\lVert a\rVert^2+\lVert b\rVert^2-2\lVert a\rVert\lVert b\rVert\cos\theta$. Expanding the left side by linearity gives $\lVert a\rVert^2+\lVert b\rVert^2-2\,a\cdot b$. Comparing the two:

$$
a\cdot b=\lVert a\rVert\,\lVert b\rVert\cos\theta
$$

Solve for the cosine and you get **cosine similarity**, which ignores length. Divide by $\lVert b\rVert$ instead and you get the **scalar projection** — the signed length of $a$'s shadow on $b$:

$$
\cos\theta=\frac{a\cdot b}{\lVert a\rVert\lVert b\rVert},\qquad \operatorname{comp}_b a=\frac{a\cdot b}{\lVert b\rVert},\qquad \operatorname{proj}_b a=\frac{a\cdot b}{b\cdot b}\,b
$$

- $\operatorname{proj}_b a$: the vector shadow of $a$ on the line through $b$

Because $|\cos\theta|\le1$, the dot product can never exceed the product of the lengths: $|a\cdot b|\le\lVert a\rVert\lVert b\rVert$ (Cauchy–Schwarz), with equality only when the vectors are parallel.

## See it

> Figure: Drag either arrow tip. At the start a = (3, 1), b = (1, 2): a·b = 5 and θ = 45°. The sign flips exactly as θ crosses 90°; the orange segment is a's projection onto b.

## Worked example

$a=(3,1)$, $b=(1,2)$. Then $a\cdot b=3\cdot1+1\cdot2=5$, $\lVert a\rVert=\sqrt{10}\approx3.162$, $\lVert b\rVert=\sqrt5\approx2.236$, so $\cos\theta=5/\sqrt{50}\approx0.7071$ and $\theta=45°$. The shadow of $a$ on $b$ has length $5/\sqrt5\approx2.236$, and the projection vector is $\tfrac{5}{5}\,b=(1,2)$ — exactly $b$.

## In code

```python
import numpy as np

a, b = np.array([3.0, 1.0]), np.array([1.0, 2.0])
dot = a @ b                                          # 5.0  (same as np.dot(a, b))
cos = dot / (np.linalg.norm(a) * np.linalg.norm(b))  # ≈ 0.7071
theta = np.degrees(np.arccos(cos))                   # ≈ 45.0
proj = (dot / (b @ b)) * b                           # array([1., 2.])

# score 3 samples (rows) against one weight vector
X = np.array([[1, 0], [0, 1], [2, 2]]); w = np.array([0.5, -1.0])
X @ w                                                # array([ 0.5, -1. , -1. ])
```

One dot product, then many at once with a matrix–vector product.

- **Neurons & linear models:** $w^\top x+b$ scores how aligned an input is with the learned weights.
- **Attention:** Scores $q\cdot k/\sqrt{d_k}$ decide how much each token reads from every other.
- **Similarity search:** Cosine similarity between embeddings ranks documents, products and images.
- **Least squares:** Regression projects $y$ onto the columns of $X$ — built entirely from dot products.

> **Big ≠ similar:** The dot product grows with length: a long vector scores high against everything roughly aligned with it. When only direction should matter — embeddings, documents of different lengths — normalize first and use $\cos\theta$.

> **High dimensions are nearly orthogonal:** Two random vectors in $\mathbb{R}^n$ have cosine close to 0, with typical size $1/\sqrt n$ — about $0.03$ for $n=1000$. So a cosine of 0.3 between embeddings is a strong signal, not a weak one.

**Question:** $a=(2,-1)$, $b=(1,2)$. What is the angle between them?

- A. 0°
- B. 45°
- C. 90°
- D. 180°

**Answer:** C. 90° — $a\cdot b=2\cdot1+(-1)\cdot2=0$, so $\cos\theta=0$ and $\theta=90°$.

**Question:** $\lVert a\rVert=2$, $\lVert b\rVert=3$ and $\theta=60°$. What is $a\cdot b$?

- A. 6
- B. 3
- C. 5.2
- D. 0

**Answer:** B. 3 — $a\cdot b=\lVert a\rVert\lVert b\rVert\cos\theta=2\cdot3\cdot\tfrac12=3$.

**Key points**

- $a\cdot b=\sum_i a_ib_i=\lVert a\rVert\lVert b\rVert\cos\theta$.
- Sign gives the angle: positive below 90°, zero exactly at 90°, negative beyond.
- Cosine similarity removes length; $a\cdot b/\lVert b\rVert$ is the projection length.
- Neurons, attention and similarity search are all dot products.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
