Definition
FlashAttention is an IO-aware, exact attention algorithm by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra and Christopher Ré, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (NeurIPS 2022). It splits , and into blocks that fit in on-chip SRAM. For each block it computes a tile of scores, its softmax contribution and its product with in one fused kernel. Tiles are merged with running maxima and normalisers (the online softmax). The matrices and are never written to GPU main memory (HBM). The backward pass recomputes them from , , and per-row statistics. The result equals standard attention up to floating-point rounding. It is not an approximation.
By 2022 many papers had proposed approximate attention (sparse, low-rank, kernel-based; see Sparse attention, Linformer, Performer) to cut the cost. Few were faster in wall-clock time than plain attention. Dao et al. argued that they counted the wrong thing. On a GPU, attention at typical head sizes is limited by memory traffic, not FLOPs. A standard implementation writes to HBM, reads it back for the softmax, writes , and reads again for . Masking and dropout add more passes. Each pass moves values through a memory that is about ten times slower than on-chip SRAM. FlashAttention removes those passes. It does more arithmetic than standard attention, because the backward pass recomputes and , and still runs faster.
The memory hierarchy the paper targets (NVIDIA A100)
- HBM
- 40–80 GB, bandwidth 1.5–2.0 TB/s
- SRAM
- 192 KB per streaming multiprocessor, 108 SMs, estimated bandwidth about 19 TB/s
- Consequence
- elementwise and reduction operations (softmax, masking, dropout) are memory-bound. With head size to 128, the intermediates dominate attention's runtime
Online softmax
- the tile of scores of the current query block against key block , shape
- per-row running maximum and running normaliser ; start at and 0
- rescales everything accumulated so far when a larger score appears. It is , so nothing overflows
- unnormalised output. After the last block is exactly , and is the row's log-sum-exp
The softmax of a row needs its maximum and its sum, which seems to require the whole row first. Milakov and Gimelshein (2018) showed that the maximum and the sum can be computed in one pass: when the running maximum grows from to , multiply the old sum by . FlashAttention applies the same correction to the weighted sum of values. Each key block is therefore used once and then discarded. Rabe and Staats (2021) used a similar chunked formulation to compute attention with memory in practice. FlashAttention adds the GPU-specific part: tile sizes chosen to fit SRAM, one fused kernel, and an IO analysis.
Tiling and IO complexity
Algorithm 1 of the paper uses block sizes and for an SRAM of values, so that , , and fit at once. The outer loop runs over key/value blocks , which are loaded once. The inner loop runs over query blocks: it loads , , , , computes the tile, updates the statistics as above, and writes , , back. Masking, softmax, dropout and the product with all happen in registers and SRAM.
- sequence length and head size
- SRAM size in values, with
- and are streamed once per key block: times values. Since (e.g. against about ), this is many times smaller than
- beyond inputs and outputs, only and (FlashAttention-2: ) per row are kept for the backward pass
The paper also proves a lower bound: no exact attention algorithm can asymptotically use fewer HBM accesses than for all in . FLOPs stay the same, : FlashAttention computes the same matrix products, only in a different order and without the round trips. With a causal mask, tiles that lie entirely above the diagonal are skipped, which saves about half the work (see Causal attention). Block-sparse FlashAttention skips every tile that a given block mask zeroes. Its IO cost is proportional to the fraction of non-zero blocks, and it was faster than all approximate methods the authors tested.
Backward pass by recomputation
- saved log-sum-exp of row ; with it, any tile of is recomputed exactly from and
- equals , the term the softmax Jacobian needs; computing it from and avoids storing
- gradient of the scores, formed tile by tile and immediately contracted into and
Standard attention saves ( values per head) for the backward pass. FlashAttention saves only and . In the backward pass it reloads , , , and block by block and recomputes each tile of and . Dropout is handled the same way: the kernel saves the random generator state and regenerates the same mask instead of storing it. This is gradient checkpointing applied inside one kernel. The extra FLOPs are cheap because the recomputation happens in SRAM, and the saved HBM reads of were the expensive part.
| metric | standard attention | FlashAttention |
|---|---|---|
| GFLOPs | 66.6 | 75.2 |
| HBM reads + writes | 40.3 GB | 4.4 GB |
| runtime | 41.7 ms | 7.3 ms |
End-to-end results reported in the paper
- BERT-large
- sequence length 512: 15% faster end-to-end training than the MLPerf 1.1 training speed record
- GPT-2
- sequence length 1K: up to 3× faster training than the HuggingFace implementation
- Long-Range Arena
- sequence lengths 1K–4K: 2.4× speed-up
- Longer context
- GPT-2 with 4K context trained faster than Megatron-LM at 1K and reached 0.7 better perplexity; 6.4 points of lift on long-document classification
- Path-X / Path-256
- first Transformers to beat chance: 61.4% at sequence length 16K and, with block-sparse FlashAttention, 63.1% at 64K
Worked example: one query, two key blocks
One query row has scores against four keys, split into two blocks of two. The values are , , , .
| after | m | ℓ | rescale e^(m_old − m) | Õ | O = Õ/ℓ |
|---|---|---|---|---|---|
| start | −∞ | 0 | (0, 0) | ||
| block 1 (s = 1, 2) | 2 | e⁻¹ + 1 = 1.3679 | 0 | (0.3679, 1.0000) | (0.2689, 0.7311) |
| block 2 (s = 3, 0) | 3 | 0.3679·1.3679 + 1 + e⁻³ = 1.5530 | e⁻¹ = 0.3679 | (2.0855, 2.4177) | (1.3429, 1.5568) |
| exact, two passes | 3 | 1.5530 | (1.3429, 1.5568) |
Step by step
- 1Block 1: tile maximum 2, , , . If the row ended here, .
- 2Block 2: tile maximum 3 exceeds 2, so and the old state is multiplied by . New terms and .
- 3 and .
- 4, identical to applied to . The saved statistic is , from which the backward pass recomputes .
1import torch, torch.nn.functional as F23def flash_forward(q, k, v, Br=64, Bc=64):4 # q, k, v: (N, d) for one head. Returns O (N, d) and log-sum-exp L (N,)5 N, d = q.shape6 O, L = torch.empty_like(q), torch.empty(N)7 for i in range(0, N, Br): # one "thread block" per query block8 qi = q[i:i + Br] * d ** -0.5 # (Br, d) kept in SRAM9 m = torch.full((qi.shape[0],), float("-inf"))10 l = torch.zeros(qi.shape[0])11 acc = torch.zeros(qi.shape[0], d) # unnormalised output12 for j in range(0, N, Bc):13 s = qi @ k[j:j + Bc].T # (Br, Bc) tile, never stored in full14 m_new = torch.maximum(m, s.max(-1).values)15 p = torch.exp(s - m_new[:, None])16 alpha = torch.exp(m - m_new) # rescales the old statistics (0 at first)17 l = alpha * l + p.sum(-1)18 acc = alpha[:, None] * acc + p @ v[j:j + Bc]19 m = m_new20 O[i:i + Br] = acc / l[:, None] # normalise once at the end21 L[i:i + Br] = m + torch.log(l) # all the backward pass needs22 return O, L2324torch.manual_seed(0)25N, d = 1000, 64 # N need not be a multiple of the block size26q, k, v = (torch.randn(N, d) for _ in range(3))27O, L = flash_forward(q, k, v)28ref = F.scaled_dot_product_attention(q[None], k[None], v[None])[0]29print(O.shape, L.shape) # torch.Size([1000, 64]) torch.Size([1000])30print(torch.allclose(O, ref, atol=1e-5)) # True3132# the backward pass rebuilds P from L instead of storing it33P = torch.softmax(q @ k.T / d ** 0.5, dim=-1) # (1000, 1000)34print(torch.allclose(P, torch.exp(q @ k.T / d ** 0.5 - L[:, None]), atol=1e-6)) # TrueThis loop shows the arithmetic, not the speed: in Python every tile is still a separate tensor in GPU memory. The speed comes from running both loops inside one CUDA kernel. In PyTorch 2.2 and later, F.scaled_dot_product_attention dispatches to a FlashAttention-2 kernel when the GPU, dtype (fp16/bf16) and head size allow it, and torch.nn.attention.sdpa_kernel(SDPBackend.FLASH_ATTENTION) forces that backend. The flash-attn package provides the authors' kernels, including variable-length batches and sliding windows.
FlashAttention-2 and FlashAttention-3
FlashAttention-2 (Tri Dao, 2023; ICLR 2024) keeps the algorithm and reorganises the work. It reduces non-matmul FLOPs: the output stays unnormalised until the end, and only is saved instead of and . It swaps the loops so the outer loop runs over query blocks. Each query block is then an independent thread block, and the kernel parallelises over sequence length as well as batch and heads, which helps when batch × heads is small. Inside a thread block it splits rather than across warps, which removes communication through shared memory. It is about 2× faster than FlashAttention, reaching 50–73% of the A100's theoretical peak FLOPs/s and up to 225 TFLOPs/s per A100 when training GPT-style models (72% model FLOPs utilisation).
FlashAttention-3 (Shah, Bikshandi, Zhang, Thakkar, Ramani and Dao, 2024; NeurIPS 2024) targets the Hopper H100, where FlashAttention-2 used only about 35% of peak. It uses the asynchronous Tensor Cores and the Tensor Memory Accelerator through warp specialisation: producer warps load tiles while consumer warps compute. "Ping-pong" scheduling between warpgroups, plus pipelining within one warpgroup, overlaps the softmax of one tile with the matrix products of another. It adds FP8 support with block quantisation and incoherent processing, a random orthogonal (Hadamard) transform that spreads outliers before quantising. In FP16 it is 1.5–2.0× faster than FlashAttention-2, up to 740 TFLOPs/s (75% utilisation). In FP8 it approaches 1.2 PFLOPs/s, with 2.6× lower numerical error than a baseline FP8 attention.
Use when
- +Exact attention; drop-in replacement with the same outputs up to rounding
- +Extra memory instead of , which enables much longer sequences
- +Several times less HBM traffic and faster wall-clock attention (5.7× in the paper's GPT-2 example)
- +Masking, dropout and causal or block-sparse tile skipping fused in the same kernel
Avoid when
- −Still FLOPs; long contexts remain quadratic in time
- −Hand-written, hardware-specific kernels; new GPUs needed a redesign (FlashAttention-3 for Hopper)
- −Restrictions on dtype and head size; arbitrary attention biases or custom score functions need kernel support
- −Recomputation adds backward FLOPs, and the default backward pass is nondeterministic
Check yourself
While processing key block j, a query row's running maximum rises from m = 2 to m = 3. What must FlashAttention do with its accumulated ℓ and Õ?
Check yourself
Which quantity does FlashAttention store for the backward pass instead of the attention matrix P?
Remember
- ✓Online softmax: , , ; at the end is exact.
- ✓IO complexity: standard attention HBM accesses, FlashAttention for SRAM size ; extra memory instead of .
- ✓The N×N matrices S and P stay in SRAM tile by tile; the backward pass recomputes them from Q, K, V and the saved log-sum-exp.
- ✓More FLOPs, much less memory traffic: in the paper's GPT-2 example 4.4 GB instead of 40.3 GB of HBM traffic and 7.3 ms instead of 41.7 ms.
- ✓FlashAttention-2 (2023): loops swapped, parallel over sequence length, about 2× faster. FlashAttention-3 (2024): Hopper asynchrony, warp specialisation and FP8, up to 740 TFLOPs/s in FP16.