GPU & TPU computing

Neural networks are mostly matrix multiplications, which thousands of parallel GPU cores and TPU systolic arrays run far faster than a CPU.

8 min read681 wordscore0 viewsupdated 2026-10-09

Definition

An accelerator (GPU or TPU) is a processor built for throughput rather than latency: it runs one simple operation, mostly multiply-accumulate, on thousands of data elements at once. Deep learning maps onto it because nearly all of its arithmetic is matrix multiplication.

Why deep learning is matrix multiplication

A dense layer applied to a mini-batch is one matrix product, Y=XW+bY = XW + b. Convolutions are lowered to matrix products (im2col or equivalent), and attention is two batched matrix products around a softmax. The backward pass is matrix products too: two per layer, one for the input gradient and one for the weight gradient.

FLOPs(XW)=2 B nin nout\text{FLOPs}(XW) = 2\,B\,n_{\text{in}}\,n_{\text{out}}
BB
batch size (rows of XX)
nin,noutn_{\text{in}}, n_{\text{out}}
input and output widths of the layer (WW is nin×noutn_{\text{in}}\times n_{\text{out}})
22
each of the B nin noutB\,n_{\text{in}}\,n_{\text{out}} multiply-accumulates is one multiply and one add

Training costs roughly 3×3\times the forward FLOPs (forward + two backward products). Every one of those multiply-adds is independent of the others within a product, which is exactly the kind of work that parallel hardware eats.

CPU, GPU, TPU

AspectCPUGPUTPU
Design goallow latency on branchy, serial codethroughput on data-parallel codethroughput on dense matrix math only
Compute unitsa few to a few dozen strong cores, wide SIMDthousands of simple lanes in SIMT warps of 32, plus tensor cores (small matrix-multiply units)one or a few large systolic arrays of MAC cells (e.g. 128×128128\times128)
Memorylarge caches, DDR main memoryhigh-bandwidth memory (HBM or GDDR), small fast on-chip SRAMHBM; operands flow cell-to-cell inside the array
ProgramminganythingCUDA / ROCm kernels; PyTorch calls libraries such as cuBLAS and cuDNNcompiled by XLA from JAX, TensorFlow or PyTorch/XLA
Native low precisionfp32/fp64, some bf16/int8tf32, fp16, bf16, int8, newer fp8bf16 multiply with fp32 accumulation, int8

A GPU hides memory latency by keeping many warps in flight and switching to a ready one whenever another waits on memory. Tensor cores go further: one instruction multiplies small tiles (such as 16×1616\times16) and adds into an accumulator, so most of a matmul's FLOPs come from them, not from the ordinary lanes.

A systolic array is a grid of multiply-accumulate (MAC) cells. Operands enter at the edges and are passed to neighbours every clock cycle, so each value read from memory is reused across a whole row or column of cells without going back to memory. That is how a TPU keeps tens of thousands of MACs busy per cycle with modest memory traffic.

▸ systolic array · 3×3 output-stationarystep 1/9
A rows →B columns ↓0c000c010c020c100c110c120c200c210c22120013201210102031
A
120013201
B
210102031
C = A·B (finished PEs)
·········
each PE: c += a·b, then a → right, b ↓ down
array: 7 cycles · one MAC unit: 27 cycles
cycle –busy 0/9MACs 0/27
  1. 1for cycle t = 0, 1, …, 3N − 3:
  2. 2 every PE (i, j) at once:
  3. 3 take a from the left, b from above
  4. 4 c[i][j] += a · b
  5. 5 pass a → right, b ↓ down
  6. 6C now sits in the PEs: shift it out

Rows of A wait on the left, columns of B on top, each skewed by one cycle so matching a[i][k] and b[k][j] meet in the same cell at the same time. Every PE's accumulator c starts at 0.

fig.A 3×3 output-stationary systolic array computing C = A·B. Rows of A (blue) enter from the left and columns of B (orange) from the top, each skewed by one cycle so that a[i][k] and b[k][j] meet in PE (i, j) at cycle i + j + k. Step through the 7 cycles: watch the diagonal wavefront of busy PEs (green), the partial sums grow, and C appear in the PEs. An N×N array finishes in 3N − 2 cycles with N³ multiply-adds; a real TPU array is about 128×128.

Memory bandwidth and arithmetic intensity

Raw FLOP/s is half the story. Every operand must come from memory, and memory is far slower than arithmetic. The deciding number for an operation is how much work it does per byte it moves.

I=FLOPsbytes moved,attainable FLOP/s=min⁡(Ppeak, I⋅BW)I=\frac{\text{FLOPs}}{\text{bytes moved}},\qquad \text{attainable FLOP/s}=\min\big(P_{\text{peak}},\ I\cdot \text{BW}\big)
II
arithmetic intensity, FLOP per byte of memory traffic
PpeakP_{\text{peak}}
peak compute throughput
BW\text{BW}
memory bandwidth, bytes per second

This is the roofline model. Below the ridge point I∗=Ppeak/BWI^\ast = P_{\text{peak}}/\text{BW} an operation is memory-bound: the cores wait for data. Above it, it is compute-bound. For current data-centre accelerators the ridge is in the tens to a few hundred FLOP/byte.

Worked example: when is a layer compute-bound?

Take an illustrative accelerator with round numbers, Ppeak=100P_{\text{peak}}=100 TFLOP/s and BW=2\text{BW}=2 TB/s, so the ridge is at I∗=50I^\ast=50 FLOP/byte. Multiply a batch by a 4096×40964096\times4096 weight matrix in 16-bit precision (2 bytes per number); bytes moved =2 (B nin+ninnout+B nout)=2\,(B\,n_{\text{in}}+n_{\text{in}}n_{\text{out}}+B\,n_{\text{out}}).

OperationFLOPsBytesI (FLOP/B)BoundAttainable
add two 2²⁰-element fp32 vectors1.0 M12.6 MB0.08memory0.17 TFLOP/s
matmul, B = 133.6 M33.6 MB1.0memory2 TFLOP/s
matmul, B = 8268 M33.7 MB8.0memory16 TFLOP/s
matmul, B = 642.15 G34.6 MB62compute100 TFLOP/s
matmul, B = 51217.2 G41.9 MB410compute100 TFLOP/s
Times assume perfect overlap; real kernels land somewhat below the roofline.
  1. 1At B=1B=1 the weights (33.6 MB) dominate the traffic and each weight is used once: about 17 µs to read them, but only 0.34 µs of arithmetic. The chip is 98% idle.
  2. 2Raising BB reuses each weight BB times for almost the same traffic, so I≈BI\approx B here. Crossing B≈50B\approx50 makes the layer compute-bound.
  3. 3Elementwise ops (add, ReLU, normalization) stay far below the ridge at any size. That is why frameworks fuse them into the neighbouring matmul instead of running them as separate passes over memory.

Precision

FormatBits (sign / exponent / mantissa)Typical use
fp321 / 8 / 23master weights, optimizer state, accumulation
tf321 / 8 / 10 (stored as 32 bits)NVIDIA tensor-core mode for fp32 matmuls
bf161 / 8 / 7training; fp32's range, so overflow is rare and no loss scaling is needed
fp161 / 5 / 10training with loss scaling (small range); inference
int88-bit integerquantized inference

Halving the bits halves memory traffic (doubling II) and, on tensor cores and TPUs, multiplies peak throughput several times over. Mixed precision multiplies in bf16/fp16, accumulates in fp32, and keeps an fp32 copy of the weights for the update.

Host ↔ device transfers

CPU RAM and accelerator memory are separate. Copies cross a link (PCIe or similar) that is typically one to two orders of magnitude slower than the accelerator's own memory. Move the model once, keep tensors on the device, and avoid per-step round-trips such as calling .item() or print on a GPU tensor inside the training loop.

python
1import time, torch
2
3device = "cuda" # needs an NVIDIA GPU
4model = torch.nn.Linear(4096, 4096).to(device) # move weights once
5x = torch.randn(512, 4096).pin_memory() # pinned host memory
6x = x.to(device, non_blocking=True) # async host->device copy
7
8with torch.autocast(device_type=device, dtype=torch.bfloat16):
9 y = model(x) # matmul runs in bf16, accumulates in fp32
10
11# Timing: CUDA launches are asynchronous, so synchronize before reading the clock.
12for _ in range(3):
13 model(x) # warm-up (kernel selection, caches)
14torch.cuda.synchronize()
15t0 = time.perf_counter()
16for _ in range(100):
17 model(x)
18torch.cuda.synchronize()
19ms = (time.perf_counter() - t0) / 100 * 1e3
20print(f"{ms:.3f} ms, {2 * 512 * 4096 * 4096 / (ms / 1e3) / 1e12:.1f} TFLOP/s")

Check yourself

A layer multiplies a batch of B inputs by a 4096×4096 weight matrix. Ignoring activations, roughly how does its arithmetic intensity change when B goes from 8 to 64?

Check yourself

In an N×N output-stationary systolic array, why is each element of A read from memory only once?

Remember

  • ✓A dense layer costs 2 B nin nout2\,B\,n_{\text{in}}\,n_{\text{out}} FLOPs; training is about 3×3\times the forward pass.
  • ✓CPUs optimize latency; GPUs run thousands of SIMT lanes plus tensor cores; TPUs stream operands through systolic arrays compiled by XLA.
  • ✓Roofline: attainable speed =min⁡(Ppeak, I⋅BW)=\min(P_{\text{peak}},\ I\cdot\text{BW}). Elementwise ops are memory-bound; large-batch matmuls are compute-bound.
  • ✓Lower precision (bf16, fp16, int8) cuts traffic and raises peak throughput; accumulate in fp32.
  • ✓Keep data on the device, batch enough work, and synchronize before timing.
← back to Deep Learning roadmap