Definition
An accelerator (GPU or TPU) is a processor built for throughput rather than latency: it runs one simple operation, mostly multiply-accumulate, on thousands of data elements at once. Deep learning maps onto it because nearly all of its arithmetic is matrix multiplication.
Why deep learning is matrix multiplication
A dense layer applied to a mini-batch is one matrix product, . Convolutions are lowered to matrix products (im2col or equivalent), and attention is two batched matrix products around a softmax. The backward pass is matrix products too: two per layer, one for the input gradient and one for the weight gradient.
- batch size (rows of )
- input and output widths of the layer ( is )
- each of the multiply-accumulates is one multiply and one add
Training costs roughly the forward FLOPs (forward + two backward products). Every one of those multiply-adds is independent of the others within a product, which is exactly the kind of work that parallel hardware eats.
CPU, GPU, TPU
| Aspect | CPU | GPU | TPU |
|---|---|---|---|
| Design goal | low latency on branchy, serial code | throughput on data-parallel code | throughput on dense matrix math only |
| Compute units | a few to a few dozen strong cores, wide SIMD | thousands of simple lanes in SIMT warps of 32, plus tensor cores (small matrix-multiply units) | one or a few large systolic arrays of MAC cells (e.g. ) |
| Memory | large caches, DDR main memory | high-bandwidth memory (HBM or GDDR), small fast on-chip SRAM | HBM; operands flow cell-to-cell inside the array |
| Programming | anything | CUDA / ROCm kernels; PyTorch calls libraries such as cuBLAS and cuDNN | compiled by XLA from JAX, TensorFlow or PyTorch/XLA |
| Native low precision | fp32/fp64, some bf16/int8 | tf32, fp16, bf16, int8, newer fp8 | bf16 multiply with fp32 accumulation, int8 |
A GPU hides memory latency by keeping many warps in flight and switching to a ready one whenever another waits on memory. Tensor cores go further: one instruction multiplies small tiles (such as ) and adds into an accumulator, so most of a matmul's FLOPs come from them, not from the ordinary lanes.
A systolic array is a grid of multiply-accumulate (MAC) cells. Operands enter at the edges and are passed to neighbours every clock cycle, so each value read from memory is reused across a whole row or column of cells without going back to memory. That is how a TPU keeps tens of thousands of MACs busy per cycle with modest memory traffic.
- 1for cycle t = 0, 1, …, 3N − 3:
- 2 every PE (i, j) at once:
- 3 take a from the left, b from above
- 4 c[i][j] += a · b
- 5 pass a → right, b ↓ down
- 6C now sits in the PEs: shift it out
Rows of A wait on the left, columns of B on top, each skewed by one cycle so matching a[i][k] and b[k][j] meet in the same cell at the same time. Every PE's accumulator c starts at 0.
Memory bandwidth and arithmetic intensity
Raw FLOP/s is half the story. Every operand must come from memory, and memory is far slower than arithmetic. The deciding number for an operation is how much work it does per byte it moves.
- arithmetic intensity, FLOP per byte of memory traffic
- peak compute throughput
- memory bandwidth, bytes per second
This is the roofline model. Below the ridge point an operation is memory-bound: the cores wait for data. Above it, it is compute-bound. For current data-centre accelerators the ridge is in the tens to a few hundred FLOP/byte.
Worked example: when is a layer compute-bound?
Take an illustrative accelerator with round numbers, TFLOP/s and TB/s, so the ridge is at FLOP/byte. Multiply a batch by a weight matrix in 16-bit precision (2 bytes per number); bytes moved .
| Operation | FLOPs | Bytes | I (FLOP/B) | Bound | Attainable |
|---|---|---|---|---|---|
| add two 2²⁰-element fp32 vectors | 1.0 M | 12.6 MB | 0.08 | memory | 0.17 TFLOP/s |
| matmul, B = 1 | 33.6 M | 33.6 MB | 1.0 | memory | 2 TFLOP/s |
| matmul, B = 8 | 268 M | 33.7 MB | 8.0 | memory | 16 TFLOP/s |
| matmul, B = 64 | 2.15 G | 34.6 MB | 62 | compute | 100 TFLOP/s |
| matmul, B = 512 | 17.2 G | 41.9 MB | 410 | compute | 100 TFLOP/s |
- 1At the weights (33.6 MB) dominate the traffic and each weight is used once: about 17 µs to read them, but only 0.34 µs of arithmetic. The chip is 98% idle.
- 2Raising reuses each weight times for almost the same traffic, so here. Crossing makes the layer compute-bound.
- 3Elementwise ops (add, ReLU, normalization) stay far below the ridge at any size. That is why frameworks fuse them into the neighbouring matmul instead of running them as separate passes over memory.
Precision
| Format | Bits (sign / exponent / mantissa) | Typical use |
|---|---|---|
| fp32 | 1 / 8 / 23 | master weights, optimizer state, accumulation |
| tf32 | 1 / 8 / 10 (stored as 32 bits) | NVIDIA tensor-core mode for fp32 matmuls |
| bf16 | 1 / 8 / 7 | training; fp32's range, so overflow is rare and no loss scaling is needed |
| fp16 | 1 / 5 / 10 | training with loss scaling (small range); inference |
| int8 | 8-bit integer | quantized inference |
Halving the bits halves memory traffic (doubling ) and, on tensor cores and TPUs, multiplies peak throughput several times over. Mixed precision multiplies in bf16/fp16, accumulates in fp32, and keeps an fp32 copy of the weights for the update.
Host ↔ device transfers
CPU RAM and accelerator memory are separate. Copies cross a link (PCIe or similar) that is typically one to two orders of magnitude slower than the accelerator's own memory. Move the model once, keep tensors on the device, and avoid per-step round-trips such as calling .item() or print on a GPU tensor inside the training loop.
1import time, torch23device = "cuda" # needs an NVIDIA GPU4model = torch.nn.Linear(4096, 4096).to(device) # move weights once5x = torch.randn(512, 4096).pin_memory() # pinned host memory6x = x.to(device, non_blocking=True) # async host->device copy78with torch.autocast(device_type=device, dtype=torch.bfloat16):9 y = model(x) # matmul runs in bf16, accumulates in fp321011# Timing: CUDA launches are asynchronous, so synchronize before reading the clock.12for _ in range(3):13 model(x) # warm-up (kernel selection, caches)14torch.cuda.synchronize()15t0 = time.perf_counter()16for _ in range(100):17 model(x)18torch.cuda.synchronize()19ms = (time.perf_counter() - t0) / 100 * 1e320print(f"{ms:.3f} ms, {2 * 512 * 4096 * 4096 / (ms / 1e3) / 1e12:.1f} TFLOP/s")Check yourself
A layer multiplies a batch of B inputs by a 4096×4096 weight matrix. Ignoring activations, roughly how does its arithmetic intensity change when B goes from 8 to 64?
Check yourself
In an N×N output-stationary systolic array, why is each element of A read from memory only once?
Remember
- ✓A dense layer costs FLOPs; training is about the forward pass.
- ✓CPUs optimize latency; GPUs run thousands of SIMT lanes plus tensor cores; TPUs stream operands through systolic arrays compiled by XLA.
- ✓Roofline: attainable speed . Elementwise ops are memory-bound; large-batch matmuls are compute-bound.
- ✓Lower precision (bf16, fp16, int8) cuts traffic and raises peak throughput; accumulate in fp32.
- ✓Keep data on the device, batch enough work, and synchronize before timing.