# GPU & TPU computing

> Neural networks are mostly matrix multiplications, which thousands of parallel GPU cores and TPU systolic arrays run far faster than a CPU.

Source: https://www.mlfoundry.in/learn/gpu-tpu-computing  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 8 min read · updated 2026-10-09

**An accelerator:** (GPU or TPU) is a processor built for throughput rather than latency: it runs one simple operation, mostly multiply-accumulate, on thousands of data elements at once. Deep learning maps onto it because nearly all of its arithmetic is matrix multiplication.

## Why deep learning is matrix multiplication

A dense layer applied to a mini-batch is one matrix product, $Y = XW + b$. Convolutions are lowered to matrix products (im2col or equivalent), and attention is two batched matrix products around a softmax. The backward pass is matrix products too: two per layer, one for the input gradient and one for the weight gradient.

$$
\text{FLOPs}(XW) = 2\,B\,n_{\text{in}}\,n_{\text{out}}
$$

- $B$: batch size (rows of $X$)
- $n_{\text{in}}, n_{\text{out}}$: input and output widths of the layer ($W$ is $n_{\text{in}}\times n_{\text{out}}$)
- $2$: each of the $B\,n_{\text{in}}\,n_{\text{out}}$ multiply-accumulates is one multiply and one add

Training costs roughly $3\times$ the forward FLOPs (forward + two backward products). Every one of those multiply-adds is independent of the others within a product, which is exactly the kind of work that parallel hardware eats.

## CPU, GPU, TPU

| Aspect | CPU | GPU | TPU |
| --- | --- | --- | --- |
| Design goal | low latency on branchy, serial code | throughput on data-parallel code | throughput on dense matrix math only |
| Compute units | a few to a few dozen strong cores, wide SIMD | thousands of simple lanes in SIMT warps of 32, plus **tensor cores** (small matrix-multiply units) | one or a few large **systolic arrays** of MAC cells (e.g. $128\times128$) |
| Memory | large caches, DDR main memory | high-bandwidth memory (HBM or GDDR), small fast on-chip SRAM | HBM; operands flow cell-to-cell inside the array |
| Programming | anything | CUDA / ROCm kernels; PyTorch calls libraries such as cuBLAS and cuDNN | compiled by **XLA** from JAX, TensorFlow or PyTorch/XLA |
| Native low precision | fp32/fp64, some bf16/int8 | tf32, fp16, bf16, int8, newer fp8 | bf16 multiply with fp32 accumulation, int8 |

A GPU hides memory latency by keeping many warps in flight and switching to a ready one whenever another waits on memory. **Tensor cores** go further: one instruction multiplies small tiles (such as $16\times16$) and adds into an accumulator, so most of a matmul's FLOPs come from them, not from the ordinary lanes.

A **systolic array** is a grid of multiply-accumulate (MAC) cells. Operands enter at the edges and are passed to neighbours every clock cycle, so each value read from memory is reused across a whole row or column of cells without going back to memory. That is how a TPU keeps tens of thousands of MACs busy per cycle with modest memory traffic.

> Figure: A 3×3 output-stationary systolic array computing C = A·B. Rows of A (blue) enter from the left and columns of B (orange) from the top, each skewed by one cycle so that a[i][k] and b[k][j] meet in PE (i, j) at cycle i + j + k. Step through the 7 cycles: watch the diagonal wavefront of busy PEs (green), the partial sums grow, and C appear in the PEs. An N×N array finishes in 3N − 2 cycles with N³ multiply-adds; a real TPU array is about 128×128.

## Memory bandwidth and arithmetic intensity

Raw FLOP/s is half the story. Every operand must come from memory, and memory is far slower than arithmetic. The deciding number for an operation is how much work it does per byte it moves.

$$
I=\frac{\text{FLOPs}}{\text{bytes moved}},\qquad \text{attainable FLOP/s}=\min\big(P_{\text{peak}},\ I\cdot \text{BW}\big)
$$

- $I$: arithmetic intensity, FLOP per byte of memory traffic
- $P_{\text{peak}}$: peak compute throughput
- $\text{BW}$: memory bandwidth, bytes per second

This is the **roofline model**. Below the ridge point $I^\ast = P_{\text{peak}}/\text{BW}$ an operation is **memory-bound**: the cores wait for data. Above it, it is **compute-bound**. For current data-centre accelerators the ridge is in the tens to a few hundred FLOP/byte.

## Worked example: when is a layer compute-bound?

Take an illustrative accelerator with round numbers, $P_{\text{peak}}=100$ TFLOP/s and $\text{BW}=2$ TB/s, so the ridge is at $I^\ast=50$ FLOP/byte. Multiply a batch by a $4096\times4096$ weight matrix in 16-bit precision (2 bytes per number); bytes moved $=2\,(B\,n_{\text{in}}+n_{\text{in}}n_{\text{out}}+B\,n_{\text{out}})$.

Times assume perfect overlap; real kernels land somewhat below the roofline.

| Operation | FLOPs | Bytes | I (FLOP/B) | Bound | Attainable |
| --- | --- | --- | --- | --- | --- |
| add two 2²⁰-element fp32 vectors | 1.0 M | 12.6 MB | 0.08 | memory | 0.17 TFLOP/s |
| matmul, B = 1 | 33.6 M | 33.6 MB | 1.0 | memory | 2 TFLOP/s |
| matmul, B = 8 | 268 M | 33.7 MB | 8.0 | memory | 16 TFLOP/s |
| matmul, B = 64 | 2.15 G | 34.6 MB | 62 | compute | 100 TFLOP/s |
| matmul, B = 512 | 17.2 G | 41.9 MB | 410 | compute | 100 TFLOP/s |

1. At $B=1$ the weights (33.6 MB) dominate the traffic and each weight is used once: about 17 µs to read them, but only 0.34 µs of arithmetic. The chip is 98% idle.
2. Raising $B$ reuses each weight $B$ times for almost the same traffic, so $I\approx B$ here. Crossing $B\approx50$ makes the layer compute-bound.
3. Elementwise ops (add, ReLU, normalization) stay far below the ridge at any size. That is why frameworks **fuse** them into the neighbouring matmul instead of running them as separate passes over memory.

## Precision

| Format | Bits (sign / exponent / mantissa) | Typical use |
| --- | --- | --- |
| fp32 | 1 / 8 / 23 | master weights, optimizer state, accumulation |
| tf32 | 1 / 8 / 10 (stored as 32 bits) | NVIDIA tensor-core mode for fp32 matmuls |
| bf16 | 1 / 8 / 7 | training; fp32's range, so overflow is rare and no loss scaling is needed |
| fp16 | 1 / 5 / 10 | training with loss scaling (small range); inference |
| int8 | 8-bit integer | quantized inference |

Halving the bits halves memory traffic (doubling $I$) and, on tensor cores and TPUs, multiplies peak throughput several times over. **Mixed precision** multiplies in bf16/fp16, accumulates in fp32, and keeps an fp32 copy of the weights for the update.

## Host ↔ device transfers

CPU RAM and accelerator memory are separate. Copies cross a link (PCIe or similar) that is typically one to two orders of magnitude slower than the accelerator's own memory. Move the model once, keep tensors on the device, and avoid per-step round-trips such as calling `.item()` or `print` on a GPU tensor inside the training loop.

```python
import time, torch

device = "cuda"                                            # needs an NVIDIA GPU
model = torch.nn.Linear(4096, 4096).to(device)            # move weights once
x = torch.randn(512, 4096).pin_memory()                    # pinned host memory
x = x.to(device, non_blocking=True)                        # async host->device copy

with torch.autocast(device_type=device, dtype=torch.bfloat16):
    y = model(x)                                           # matmul runs in bf16, accumulates in fp32

# Timing: CUDA launches are asynchronous, so synchronize before reading the clock.
for _ in range(3):
    model(x)                                               # warm-up (kernel selection, caches)
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(100):
    model(x)
torch.cuda.synchronize()
ms = (time.perf_counter() - t0) / 100 * 1e3
print(f"{ms:.3f} ms, {2 * 512 * 4096 * 4096 / (ms / 1e3) / 1e12:.1f} TFLOP/s")
```

> **Timing without synchronizing:** A CUDA kernel launch returns immediately; the GPU runs it later. Without `torch.cuda.synchronize()` before each clock read, you time the launch, not the work, and get absurdly fast numbers. Always warm up first, too.

> **Small batches waste the hardware:** At batch size 1 a dense layer is memory-bound regardless of how many cores the chip has. Batching (or, for inference, serving many requests together) is what turns bandwidth-limited work into compute-limited work.

**Question:** A layer multiplies a batch of B inputs by a 4096×4096 weight matrix. Ignoring activations, roughly how does its arithmetic intensity change when B goes from 8 to 64?

- A. It stays the same
- B. It grows about 8×
- C. It grows about 64×
- D. It shrinks, because more data is moved

**Answer:** B. It grows about 8× — FLOPs grow with $B$ while traffic is dominated by the fixed 16.8 M weights, so $I\approx B$: 8.0 → 62 FLOP/byte in the worked example, almost 8×.

**Question:** In an N×N output-stationary systolic array, why is each element of A read from memory only once?

- A. Because A is stored in the array's registers before the computation
- B. Because each value is passed from cell to cell along its row and reused by all N cells it meets
- C. Because the array only uses each element of A once in the product
- D. Because B is broadcast to all cells at once

**Answer:** B. Because each value is passed from cell to cell along its row and reused by all N cells it meets — $a_{ik}$ enters at the left edge and moves right one cell per cycle, contributing to $c_{i0},\dots,c_{i,N-1}$ on the way. Reuse through neighbour-to-neighbour passing is what keeps memory traffic low.

**Key points**

- A dense layer costs $2\,B\,n_{\text{in}}\,n_{\text{out}}$ FLOPs; training is about $3\times$ the forward pass.
- CPUs optimize latency; GPUs run thousands of SIMT lanes plus tensor cores; TPUs stream operands through systolic arrays compiled by XLA.
- Roofline: attainable speed $=\min(P_{\text{peak}},\ I\cdot\text{BW})$. Elementwise ops are memory-bound; large-batch matmuls are compute-bound.
- Lower precision (bf16, fp16, int8) cuts traffic and raises peak throughput; accumulate in fp32.
- Keep data on the device, batch enough work, and synchronize before timing.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
