Vectors & the Dot Product

a·b = ‖a‖‖b‖cos θ — the dot product measures alignment, and it is the core operation of every neural layer.

5 min read385 wordscore14 viewsupdated 2026-10-03

Definition

The dot product of two vectors a,b∈Rna, b\in\mathbb{R}^n is the number a⋅b=∑i=1naibia\cdot b=\sum_{i=1}^n a_ib_i — multiply matching coordinates and add. Geometrically it equals ∥a∥ ∥b∥cos⁡θ\lVert a\rVert\,\lVert b\rVert\cos\theta, where θ\theta is the angle between them.

At a glance

Notation
a⋅ba\cdot b, a⊤ba^\top b, ⟨a,b⟩\langle a, b\rangle
Input → output
two vectors of the same length → one number
Cost
nn multiplications, n−1n-1 additions
Key property
a⋅b=0a\cdot b=0 ⇔ orthogonal (for nonzero vectors); a⋅a=∥a∥2a\cdot a=\lVert a\rVert^2
Prerequisites
coordinates, Pythagoras, cosine
Used in
neurons, attention, cosine similarity, projections, every matrix product

The intuition

A vector is an arrow from the origin — or, equally, a list of nn numbers such as one row of a dataset. The dot product asks how much two arrows point the same way, weighted by their lengths. Same direction gives a large positive number, perpendicular gives exactly zero, opposite gives a large negative number.

Read as arithmetic, w⋅xw\cdot x is a weighted sum: each feature xix_i times its importance wiw_i. That single operation is a linear model, one neuron, and one entry of every matrix product.

The math

a⋅b=∑i=1naibi,∥a∥=a⋅aa\cdot b=\sum_{i=1}^{n}a_ib_i,\qquad \lVert a\rVert=\sqrt{a\cdot a}
aia_i
ii-th coordinate of aa
∥a∥\lVert a\rVert
Euclidean length of a

Directly from the sum: it is symmetric (a⋅b=b⋅aa\cdot b=b\cdot a), linear in each argument ((αa+c)⋅b=α a⋅b+c⋅b(\alpha a+c)\cdot b=\alpha\,a\cdot b+c\cdot b), and a⋅a≥0a\cdot a\ge0 with equality only for a=0a=0.

Why the cosine. The triangle with sides aa, bb and a−ba-b obeys the law of cosines, ∥a−b∥2=∥a∥2+∥b∥2−2∥a∥∥b∥cos⁡θ\lVert a-b\rVert^2=\lVert a\rVert^2+\lVert b\rVert^2-2\lVert a\rVert\lVert b\rVert\cos\theta. Expanding the left side by linearity gives ∥a∥2+∥b∥2−2 a⋅b\lVert a\rVert^2+\lVert b\rVert^2-2\,a\cdot b. Comparing the two:

a⋅b=∥a∥ ∥b∥cos⁡θa\cdot b=\lVert a\rVert\,\lVert b\rVert\cos\theta

Solve for the cosine and you get cosine similarity, which ignores length. Divide by ∥b∥\lVert b\rVert instead and you get the scalar projection — the signed length of aa's shadow on bb:

cos⁡θ=a⋅b∥a∥∥b∥,comp⁡ba=a⋅b∥b∥,proj⁡ba=a⋅bb⋅b b\cos\theta=\frac{a\cdot b}{\lVert a\rVert\lVert b\rVert},\qquad \operatorname{comp}_b a=\frac{a\cdot b}{\lVert b\rVert},\qquad \operatorname{proj}_b a=\frac{a\cdot b}{b\cdot b}\,b
proj⁡ba\operatorname{proj}_b a
the vector shadow of aa on the line through bb

Because ∣cos⁡θ∣≤1|\cos\theta|\le1, the dot product can never exceed the product of the lengths: ∣a⋅b∣≤∥a∥∥b∥|a\cdot b|\le\lVert a\rVert\lVert b\rVert (Cauchy–Schwarz), with equality only when the vectors are parallel.

See it

ba
a = (3, 1)b = (1, 2)a·b = 5.00θ = 45.0°
fig.Drag either arrow tip. At the start a = (3, 1), b = (1, 2): a·b = 5 and θ = 45°. The sign flips exactly as θ crosses 90°; the orange segment is a's projection onto b.

Worked example

a=(3,1)a=(3,1), b=(1,2)b=(1,2). Then a⋅b=3⋅1+1⋅2=5a\cdot b=3\cdot1+1\cdot2=5, ∥a∥=10≈3.162\lVert a\rVert=\sqrt{10}\approx3.162, ∥b∥=5≈2.236\lVert b\rVert=\sqrt5\approx2.236, so cos⁡θ=5/50≈0.7071\cos\theta=5/\sqrt{50}\approx0.7071 and θ=45°\theta=45°. The shadow of aa on bb has length 5/5≈2.2365/\sqrt5\approx2.236, and the projection vector is 55 b=(1,2)\tfrac{5}{5}\,b=(1,2) — exactly bb.

In code

python
1import numpy as np
2
3a, b = np.array([3.0, 1.0]), np.array([1.0, 2.0])
4dot = a @ b # 5.0 (same as np.dot(a, b))
5cos = dot / (np.linalg.norm(a) * np.linalg.norm(b)) # ≈ 0.7071
6theta = np.degrees(np.arccos(cos)) # ≈ 45.0
7proj = (dot / (b @ b)) * b # array([1., 2.])
8
9# score 3 samples (rows) against one weight vector
10X = np.array([[1, 0], [0, 1], [2, 2]]); w = np.array([0.5, -1.0])
11X @ w # array([ 0.5, -1. , -1. ])
One dot product, then many at once with a matrix–vector product.

Where it's used

Neurons & linear models

w⊤x+bw^\top x+b scores how aligned an input is with the learned weights.

Attention

Scores q⋅k/dkq\cdot k/\sqrt{d_k} decide how much each token reads from every other.

Similarity search

Cosine similarity between embeddings ranks documents, products and images.

Least squares

Regression projects yy onto the columns of XX — built entirely from dot products.

Check yourself

a=(2,−1)a=(2,-1), b=(1,2)b=(1,2). What is the angle between them?

Check yourself

∥a∥=2\lVert a\rVert=2, ∥b∥=3\lVert b\rVert=3 and θ=60°\theta=60°. What is a⋅ba\cdot b?

Remember

  • ✓a⋅b=∑iaibi=∥a∥∥b∥cos⁡θa\cdot b=\sum_i a_ib_i=\lVert a\rVert\lVert b\rVert\cos\theta.
  • ✓Sign gives the angle: positive below 90°, zero exactly at 90°, negative beyond.
  • ✓Cosine similarity removes length; a⋅b/∥b∥a\cdot b/\lVert b\rVert is the projection length.
  • ✓Neurons, attention and similarity search are all dot products.
← back to Mathematics roadmap