FlashAttention

FlashAttention computes exact attention tile by tile in on-chip SRAM, combining tiles with an online softmax, so the N×N score matrix is never written to GPU memory. The backward pass recomputes it from saved row statistics. HBM traffic falls from Θ(Nd + N²) to Θ(N²d²/M), and extra memory is O(N).

15 min read1307 wordsadvanced0 viewsupdated 2026-10-12

Definition

FlashAttention is an IO-aware, exact attention algorithm by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra and Christopher Ré, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (NeurIPS 2022). It splits QQ, KK and VV into blocks that fit in on-chip SRAM. For each block it computes a tile of scores, its softmax contribution and its product with VV in one fused kernel. Tiles are merged with running maxima and normalisers (the online softmax). The N×NN\times N matrices S=QK⊤S=QK^\top and P=softmax(S)P=\mathrm{softmax}(S) are never written to GPU main memory (HBM). The backward pass recomputes them from QQ, KK, VV and per-row statistics. The result equals standard attention up to floating-point rounding. It is not an approximation.

By 2022 many papers had proposed approximate attention (sparse, low-rank, kernel-based; see Sparse attention, Linformer, Performer) to cut the O(N2)O(N^2) cost. Few were faster in wall-clock time than plain attention. Dao et al. argued that they counted the wrong thing. On a GPU, attention at typical head sizes is limited by memory traffic, not FLOPs. A standard implementation writes SS to HBM, reads it back for the softmax, writes PP, and reads PP again for PVPV. Masking and dropout add more passes. Each pass moves N2N^2 values through a memory that is about ten times slower than on-chip SRAM. FlashAttention removes those passes. It does more arithmetic than standard attention, because the backward pass recomputes SS and PP, and still runs faster.

The memory hierarchy the paper targets (NVIDIA A100)

HBM
40–80 GB, bandwidth 1.5–2.0 TB/s
SRAM
192 KB per streaming multiprocessor, 108 SMs, estimated bandwidth about 19 TB/s
Consequence
elementwise and reduction operations (softmax, masking, dropout) are memory-bound. With head size d=64d=64 to 128, the N2N^2 intermediates dominate attention's runtime

Online softmax

m(j)=max⁡ ⁣(m(j−1), rowmax(S:,j)),ℓ(j)=em(j−1)−m(j)ℓ(j−1)+rowsum(eS:,j−m(j)),O~(j)=em(j−1)−m(j) O~(j−1)+eS:,j−m(j)Vjm^{(j)}=\max\!\big(m^{(j-1)},\,\mathrm{rowmax}(S_{:,j})\big),\qquad \ell^{(j)}=e^{m^{(j-1)}-m^{(j)}}\ell^{(j-1)}+\mathrm{rowsum}\big(e^{S_{:,j}-m^{(j)}}\big),\qquad \tilde O^{(j)}=e^{m^{(j-1)}-m^{(j)}}\,\tilde O^{(j-1)}+e^{S_{:,j}-m^{(j)}}V_j
S:,jS_{:,j}
the tile of scores QiKj⊤/dQ_iK_j^\top/\sqrt d of the current query block against key block jj, shape Br×BcB_r\times B_c
m, ℓm,\ \ell
per-row running maximum and running normaliser ∑es−m\sum e^{s-m}; start at −∞-\infty and 0
em(j−1)−m(j)e^{m^{(j-1)}-m^{(j)}}
rescales everything accumulated so far when a larger score appears. It is ≤1\le1, so nothing overflows
O~\tilde O
unnormalised output. After the last block O=O~/ℓO=\tilde O/\ell is exactly softmax(S)V\mathrm{softmax}(S)V, and L=m+log⁡ℓL=m+\log\ell is the row's log-sum-exp

The softmax of a row needs its maximum and its sum, which seems to require the whole row first. Milakov and Gimelshein (2018) showed that the maximum and the sum can be computed in one pass: when the running maximum grows from mm to m′m', multiply the old sum by em−m′e^{m-m'}. FlashAttention applies the same correction to the weighted sum of values. Each key block is therefore used once and then discarded. Rabe and Staats (2021) used a similar chunked formulation to compute attention with O(N)O(\sqrt N) memory in practice. FlashAttention adds the GPU-specific part: tile sizes chosen to fit SRAM, one fused kernel, and an IO analysis.

Tiling and IO complexity

Algorithm 1 of the paper uses block sizes Bc=⌈M/4d⌉B_c=\lceil M/4d\rceil and Br=min⁡(⌈M/4d⌉,d)B_r=\min(\lceil M/4d\rceil,d) for an SRAM of MM values, so that QiQ_i, KjK_j, VjV_j and OiO_i fit at once. The outer loop runs over key/value blocks Kj,VjK_j,V_j, which are loaded once. The inner loop runs over query blocks: it loads QiQ_i, OiO_i, ℓi\ell_i, mim_i, computes the tile, updates the statistics as above, and writes OiO_i, ℓi\ell_i, mim_i back. Masking, softmax, dropout and the product with VV all happen in registers and SRAM.

standard:  Θ(Nd+N2) HBM accesses,FlashAttention:  Θ ⁣(N2d2M),extra memory: O(N2) → O(N)\text{standard: }\ \Theta\big(Nd+N^2\big)\ \text{HBM accesses},\qquad \text{FlashAttention: }\ \Theta\!\Big(\frac{N^2d^2}{M}\Big),\qquad \text{extra memory: } O(N^2)\ \to\ O(N)
N, dN,\ d
sequence length and head size
MM
SRAM size in values, with d≤M≤Ndd\le M\le Nd
N2d2/MN^2d^2/M
QQ and OO are streamed once per key block: Tc=N/Bc≈4Nd/MT_c=N/B_c\approx 4Nd/M times NdNd values. Since d2≪Md^2\ll M (e.g. 642=409664^2=4096 against about 10510^5), this is many times smaller than N2N^2
O(N)O(N)
beyond inputs and outputs, only mm and ℓ\ell (FlashAttention-2: LL) per row are kept for the backward pass

The paper also proves a lower bound: no exact attention algorithm can asymptotically use fewer HBM accesses than Θ(N2d2/M)\Theta(N^2d^2/M) for all MM in [d,Nd][d,Nd]. FLOPs stay the same, O(N2d)O(N^2d): FlashAttention computes the same matrix products, only in a different order and without the round trips. With a causal mask, tiles that lie entirely above the diagonal are skipped, which saves about half the work (see Causal attention). Block-sparse FlashAttention skips every tile that a given block mask zeroes. Its IO cost is proportional to the fraction of non-zero blocks, and it was faster than all approximate methods the authors tested.

Backward pass by recomputation

Pij=exp⁡ ⁣(Sij−Li),dV=P⊤dO,dP=dO V⊤,Di=rowsum(dOi∘Oi),dS=P∘(dP−D),dQ=dS Kd,dK=dS⊤QdP_{ij}=\exp\!\big(S_{ij}-L_i\big),\qquad dV=P^\top dO,\qquad dP=dO\,V^\top,\qquad D_i=\mathrm{rowsum}(dO_i\circ O_i),\qquad dS=P\circ(dP-D),\qquad dQ=\frac{dS\,K}{\sqrt d},\quad dK=\frac{dS^\top Q}{\sqrt d}
Li=mi+log⁡ℓiL_i=m_i+\log\ell_i
saved log-sum-exp of row ii; with it, any tile of PP is recomputed exactly from QiQ_i and KjK_j
DiD_i
equals ∑jPij dPij\sum_j P_{ij}\,dP_{ij}, the term the softmax Jacobian needs; computing it from dOdO and OO avoids storing PP
dSdS
gradient of the scores, formed tile by tile and immediately contracted into dQdQ and dKdK

Standard attention saves PP (N2N^2 values per head) for the backward pass. FlashAttention saves only OO and LL. In the backward pass it reloads QQ, KK, VV, OO and dOdO block by block and recomputes each tile of SS and PP. Dropout is handled the same way: the kernel saves the random generator state and regenerates the same mask instead of storing it. This is gradient checkpointing applied inside one kernel. The extra FLOPs are cheap because the recomputation happens in SRAM, and the saved HBM reads of PP were the expensive part.

metricstandard attentionFlashAttention
GFLOPs66.675.2
HBM reads + writes40.3 GB4.4 GB
runtime41.7 ms7.3 ms
GPT-2 medium attention on an A100, forward + backward (seq. length 1024, head size 64, 16 heads, batch 64; Dao et al. 2022, Fig. 2)

End-to-end results reported in the paper

BERT-large
sequence length 512: 15% faster end-to-end training than the MLPerf 1.1 training speed record
GPT-2
sequence length 1K: up to 3× faster training than the HuggingFace implementation
Long-Range Arena
sequence lengths 1K–4K: 2.4× speed-up
Longer context
GPT-2 with 4K context trained faster than Megatron-LM at 1K and reached 0.7 better perplexity; 6.4 points of lift on long-document classification
Path-X / Path-256
first Transformers to beat chance: 61.4% at sequence length 16K and, with block-sparse FlashAttention, 63.1% at 64K
S = QKᵀ/√d is computed tile by tile in SRAM and never written to HBMkeys j → (K_j, V_j blocks of 4) · click a rowqueries i (Q_i blocks)shade = exp(s − max) · □ current tilerow 10: 3/4 key blocks seenm (running max) 2.657 exact 2.657ℓ (running Σ e^(s−m)) 2.166 exact 3.204last rescale e^(m_old − m_new) 0.226max |O − exact| 3.6e-1O per dim: ■ running ■ exacttoy: N = 16, d = 4, tiles 4×4tile 11 of 16 (Q_2, K_2)HBM accesses so far 800FlashAttention total 1152standard total 1280 (S, P written + read)at N = 16 the N² terms are tiny; scale up belowone head, N = 4096, d = 64, fp16SRAM M = 64K values → B_c = 256, B_r = 64, T_c = 16standard HBM traffic 130.0 MiBFlashAttention HBM 25.5 MiBfewer HBM accesses 5.10×kept for backward, standard P: 32.0 MiBkept for backward, Flash m, ℓ: 16 KiBforward matmul FLOPs same 4N²d, exact resultstandard 4Nd + 4N² · Flash 2Nd + T_c(3Nd + 4N)
tile sizehead size d
fig.A 16×16 score matrix (d = 4, random Q, K, V) processed in 4×4 tiles in the order of Algorithm 1: for each K/V block, every Q block. Row 10 lies in Q block 2. After the tile with K block 0 its running max is m = 1.172 and ℓ = 2.732, and its output is still far from the exact one (max error 1.4). At tile 11, K block 2 contains a larger score: m jumps to 2.657, and the old ℓ and O are multiplied by e^(1.172 − 2.657) = 0.226. After tile 15 the row matches exact softmax attention to 1.1e-16, with m = 2.657 and ℓ = 3.204 identical to the two-pass values. The toy counters count HBM values read and written. With 4×4 tiles FlashAttention moves 1152 values, against 1280 for standard attention, which writes and rereads S and P. With 2×2 tiles it moves 2176, with 8×8 only 640: larger tiles mean fewer reloads of Q and O. The lower right applies the same IO model at full size, with an SRAM of 64K values. At N = 4096 and d = 64 it gives 130.0 MiB against 25.5 MiB per head (5.10× fewer accesses), and at N = 16384 it gives 2.01 GiB against 396 MiB. The ratio is about M/3d², so it drops to 1.33× at d = 128. The real gain is larger, because standard implementations also make extra N² passes for masking and dropout. For the backward pass, standard attention keeps P (32 MiB at N = 4096); FlashAttention keeps 16 KiB of row statistics.

Worked example: one query, two key blocks

One query row has scores s=(1, 2 ∣ 3, 0)s=(1,\,2\,|\,3,\,0) against four keys, split into two blocks of two. The values are v1=(1,0)v_1=(1,0), v2=(0,1)v_2=(0,1), v3=(2,2)v_3=(2,2), v4=(−1,1)v_4=(-1,1).

aftermℓrescale e^(m_old − m)ÕO = Õ/ℓ
start−∞0(0, 0)
block 1 (s = 1, 2)2e⁻¹ + 1 = 1.36790(0.3679, 1.0000)(0.2689, 0.7311)
block 2 (s = 3, 0)30.3679·1.3679 + 1 + e⁻³ = 1.5530e⁻¹ = 0.3679(2.0855, 2.4177)(1.3429, 1.5568)
exact, two passes31.5530(1.3429, 1.5568)
Online-softmax state after each block (computed in Node); Õ is the unnormalised output

Step by step

  1. 1Block 1: tile maximum 2, p=(e−1,e0)=(0.3679,1)p=(e^{-1},e^{0})=(0.3679,1), ℓ=1.3679\ell=1.3679, O~=0.3679 v1+1⋅v2=(0.3679, 1)\tilde O=0.3679\,v_1+1\cdot v_2=(0.3679,\,1). If the row ended here, O=(0.2689, 0.7311)O=(0.2689,\,0.7311).
  2. 2Block 2: tile maximum 3 exceeds 2, so m=3m=3 and the old state is multiplied by e2−3=0.3679e^{2-3}=0.3679. New terms e3−3=1e^{3-3}=1 and e0−3=0.0498e^{0-3}=0.0498.
  3. 3ℓ=0.3679⋅1.3679+1+0.0498=1.5530\ell=0.3679\cdot1.3679+1+0.0498=1.5530 and O~=0.3679 (0.3679,1)+1⋅(2,2)+0.0498 (−1,1)=(2.0855, 2.4177)\tilde O=0.3679\,(0.3679,1)+1\cdot(2,2)+0.0498\,(-1,1)=(2.0855,\,2.4177).
  4. 4O=O~/ℓ=(1.3429, 1.5568)O=\tilde O/\ell=(1.3429,\,1.5568), identical to softmax(1,2,3,0)=(0.0871, 0.2369, 0.6439, 0.0321)\mathrm{softmax}(1,2,3,0)=(0.0871,\,0.2369,\,0.6439,\,0.0321) applied to VV. The saved statistic is L=3+log⁡1.5530=3.4402L=3+\log1.5530=3.4402, from which the backward pass recomputes P1j=esj−3.4402P_{1j}=e^{s_j-3.4402}.
python
1import torch, torch.nn.functional as F
2
3def flash_forward(q, k, v, Br=64, Bc=64):
4 # q, k, v: (N, d) for one head. Returns O (N, d) and log-sum-exp L (N,)
5 N, d = q.shape
6 O, L = torch.empty_like(q), torch.empty(N)
7 for i in range(0, N, Br): # one "thread block" per query block
8 qi = q[i:i + Br] * d ** -0.5 # (Br, d) kept in SRAM
9 m = torch.full((qi.shape[0],), float("-inf"))
10 l = torch.zeros(qi.shape[0])
11 acc = torch.zeros(qi.shape[0], d) # unnormalised output
12 for j in range(0, N, Bc):
13 s = qi @ k[j:j + Bc].T # (Br, Bc) tile, never stored in full
14 m_new = torch.maximum(m, s.max(-1).values)
15 p = torch.exp(s - m_new[:, None])
16 alpha = torch.exp(m - m_new) # rescales the old statistics (0 at first)
17 l = alpha * l + p.sum(-1)
18 acc = alpha[:, None] * acc + p @ v[j:j + Bc]
19 m = m_new
20 O[i:i + Br] = acc / l[:, None] # normalise once at the end
21 L[i:i + Br] = m + torch.log(l) # all the backward pass needs
22 return O, L
23
24torch.manual_seed(0)
25N, d = 1000, 64 # N need not be a multiple of the block size
26q, k, v = (torch.randn(N, d) for _ in range(3))
27O, L = flash_forward(q, k, v)
28ref = F.scaled_dot_product_attention(q[None], k[None], v[None])[0]
29print(O.shape, L.shape) # torch.Size([1000, 64]) torch.Size([1000])
30print(torch.allclose(O, ref, atol=1e-5)) # True
31
32# the backward pass rebuilds P from L instead of storing it
33P = torch.softmax(q @ k.T / d ** 0.5, dim=-1) # (1000, 1000)
34print(torch.allclose(P, torch.exp(q @ k.T / d ** 0.5 - L[:, None]), atol=1e-6)) # True
Reference tiled forward pass with online softmax (FlashAttention-2 loop order), checked against PyTorch SDPA

This loop shows the arithmetic, not the speed: in Python every tile is still a separate tensor in GPU memory. The speed comes from running both loops inside one CUDA kernel. In PyTorch 2.2 and later, F.scaled_dot_product_attention dispatches to a FlashAttention-2 kernel when the GPU, dtype (fp16/bf16) and head size allow it, and torch.nn.attention.sdpa_kernel(SDPBackend.FLASH_ATTENTION) forces that backend. The flash-attn package provides the authors' kernels, including variable-length batches and sliding windows.

FlashAttention-2 and FlashAttention-3

FlashAttention-2 (Tri Dao, 2023; ICLR 2024) keeps the algorithm and reorganises the work. It reduces non-matmul FLOPs: the output stays unnormalised until the end, and only LL is saved instead of mm and ℓ\ell. It swaps the loops so the outer loop runs over query blocks. Each query block is then an independent thread block, and the kernel parallelises over sequence length as well as batch and heads, which helps when batch × heads is small. Inside a thread block it splits QQ rather than KK across warps, which removes communication through shared memory. It is about 2× faster than FlashAttention, reaching 50–73% of the A100's theoretical peak FLOPs/s and up to 225 TFLOPs/s per A100 when training GPT-style models (72% model FLOPs utilisation).

FlashAttention-3 (Shah, Bikshandi, Zhang, Thakkar, Ramani and Dao, 2024; NeurIPS 2024) targets the Hopper H100, where FlashAttention-2 used only about 35% of peak. It uses the asynchronous Tensor Cores and the Tensor Memory Accelerator through warp specialisation: producer warps load tiles while consumer warps compute. "Ping-pong" scheduling between warpgroups, plus pipelining within one warpgroup, overlaps the softmax of one tile with the matrix products of another. It adds FP8 support with block quantisation and incoherent processing, a random orthogonal (Hadamard) transform that spreads outliers before quantising. In FP16 it is 1.5–2.0× faster than FlashAttention-2, up to 740 TFLOPs/s (75% utilisation). In FP8 it approaches 1.2 PFLOPs/s, with 2.6× lower numerical error than a baseline FP8 attention.

Use when

  • +Exact attention; drop-in replacement with the same outputs up to rounding
  • +Extra memory O(N)O(N) instead of O(N2)O(N^2), which enables much longer sequences
  • +Several times less HBM traffic and faster wall-clock attention (5.7× in the paper's GPT-2 example)
  • +Masking, dropout and causal or block-sparse tile skipping fused in the same kernel

Avoid when

  • −Still O(N2d)O(N^2d) FLOPs; long contexts remain quadratic in time
  • −Hand-written, hardware-specific kernels; new GPUs needed a redesign (FlashAttention-3 for Hopper)
  • −Restrictions on dtype and head size; arbitrary attention biases or custom score functions need kernel support
  • −Recomputation adds backward FLOPs, and the default backward pass is nondeterministic

Check yourself

While processing key block j, a query row's running maximum rises from m = 2 to m = 3. What must FlashAttention do with its accumulated ℓ and Õ?

Check yourself

Which quantity does FlashAttention store for the backward pass instead of the attention matrix P?

Remember

  • ✓Online softmax: m′=max⁡(m,rowmax Sj)m'=\max(m,\mathrm{rowmax}\,S_j), ℓ′=em−m′ℓ+rowsum eSj−m′\ell'=e^{m-m'}\ell+\mathrm{rowsum}\,e^{S_j-m'}, O~′=em−m′O~+eSj−m′Vj\tilde O'=e^{m-m'}\tilde O+e^{S_j-m'}V_j; at the end O=O~/ℓO=\tilde O/\ell is exact.
  • ✓IO complexity: standard attention Θ(Nd+N2)\Theta(Nd+N^2) HBM accesses, FlashAttention Θ(N2d2/M)\Theta(N^2d^2/M) for SRAM size MM; extra memory O(N)O(N) instead of O(N2)O(N^2).
  • ✓The N×N matrices S and P stay in SRAM tile by tile; the backward pass recomputes them from Q, K, V and the saved log-sum-exp.
  • ✓More FLOPs, much less memory traffic: in the paper's GPT-2 example 4.4 GB instead of 40.3 GB of HBM traffic and 7.3 ms instead of 41.7 ms.
  • ✓FlashAttention-2 (2023): loops swapped, parallel over sequence length, about 2× faster. FlashAttention-3 (2024): Hopper asynchrony, warp specialisation and FP8, up to 740 TFLOPs/s in FP16.
← back to Deep Learning roadmap