# FlashAttention

> FlashAttention computes exact attention tile by tile in on-chip SRAM, combining tiles with an online softmax, so the N×N score matrix is never written to GPU memory. The backward pass recomputes it from saved row statistics. HBM traffic falls from Θ(Nd + N²) to Θ(N²d²/M), and extra memory is O(N).

Source: https://www.mlfoundry.in/learn/flashattention  
Section: Deep Learning › Attention and Transformer Architectures  
Level: advanced · 15 min read · updated 2026-10-12

**FlashAttention:** is an **IO-aware**, **exact** attention algorithm by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra and Christopher Ré, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (NeurIPS 2022). It splits $Q$, $K$ and $V$ into blocks that fit in on-chip SRAM. For each block it computes a tile of scores, its softmax contribution and its product with $V$ in one fused kernel. Tiles are merged with running maxima and normalisers (the **online softmax**). The $N\times N$ matrices $S=QK^\top$ and $P=\mathrm{softmax}(S)$ are never written to GPU main memory (HBM). The backward pass recomputes them from $Q$, $K$, $V$ and per-row statistics. The result equals standard attention up to floating-point rounding. It is not an approximation.

By 2022 many papers had proposed approximate attention (sparse, low-rank, kernel-based; see **Sparse attention**, **Linformer**, **Performer**) to cut the $O(N^2)$ cost. Few were faster in wall-clock time than plain attention. Dao et al. argued that they counted the wrong thing. On a GPU, attention at typical head sizes is limited by **memory traffic**, not FLOPs. A standard implementation writes $S$ to HBM, reads it back for the softmax, writes $P$, and reads $P$ again for $PV$. Masking and dropout add more passes. Each pass moves $N^2$ values through a memory that is about ten times slower than on-chip SRAM. FlashAttention removes those passes. It does more arithmetic than standard attention, because the backward pass recomputes $S$ and $P$, and still runs faster.

**The memory hierarchy the paper targets (NVIDIA A100)**

- **HBM:** 40–80 GB, bandwidth 1.5–2.0 TB/s
- **SRAM:** 192 KB per streaming multiprocessor, 108 SMs, estimated bandwidth about 19 TB/s
- **Consequence:** elementwise and reduction operations (softmax, masking, dropout) are **memory-bound**. With head size $d=64$ to 128, the $N^2$ intermediates dominate attention's runtime

## Online softmax

$$
m^{(j)}=\max\!\big(m^{(j-1)},\,\mathrm{rowmax}(S_{:,j})\big),\qquad \ell^{(j)}=e^{m^{(j-1)}-m^{(j)}}\ell^{(j-1)}+\mathrm{rowsum}\big(e^{S_{:,j}-m^{(j)}}\big),\qquad \tilde O^{(j)}=e^{m^{(j-1)}-m^{(j)}}\,\tilde O^{(j-1)}+e^{S_{:,j}-m^{(j)}}V_j
$$

- $S_{:,j}$: the tile of scores $Q_iK_j^\top/\sqrt d$ of the current query block against key block $j$, shape $B_r\times B_c$
- $m,\ \ell$: per-row running maximum and running normaliser $\sum e^{s-m}$; start at $-\infty$ and 0
- $e^{m^{(j-1)}-m^{(j)}}$: rescales everything accumulated so far when a larger score appears. It is $\le1$, so nothing overflows
- $\tilde O$: unnormalised output. After the last block $O=\tilde O/\ell$ is exactly $\mathrm{softmax}(S)V$, and $L=m+\log\ell$ is the row's log-sum-exp

The softmax of a row needs its maximum and its sum, which seems to require the whole row first. Milakov and Gimelshein (2018) showed that the maximum and the sum can be computed in one pass: when the running maximum grows from $m$ to $m'$, multiply the old sum by $e^{m-m'}$. FlashAttention applies the same correction to the weighted sum of values. Each key block is therefore used once and then discarded. Rabe and Staats (2021) used a similar chunked formulation to compute attention with $O(\sqrt N)$ memory in practice. FlashAttention adds the GPU-specific part: tile sizes chosen to fit SRAM, one fused kernel, and an IO analysis.

## Tiling and IO complexity

Algorithm 1 of the paper uses block sizes $B_c=\lceil M/4d\rceil$ and $B_r=\min(\lceil M/4d\rceil,d)$ for an SRAM of $M$ values, so that $Q_i$, $K_j$, $V_j$ and $O_i$ fit at once. The **outer** loop runs over key/value blocks $K_j,V_j$, which are loaded once. The **inner** loop runs over query blocks: it loads $Q_i$, $O_i$, $\ell_i$, $m_i$, computes the tile, updates the statistics as above, and writes $O_i$, $\ell_i$, $m_i$ back. Masking, softmax, dropout and the product with $V$ all happen in registers and SRAM.

$$
\text{standard: }\ \Theta\big(Nd+N^2\big)\ \text{HBM accesses},\qquad \text{FlashAttention: }\ \Theta\!\Big(\frac{N^2d^2}{M}\Big),\qquad \text{extra memory: } O(N^2)\ \to\ O(N)
$$

- $N,\ d$: sequence length and head size
- $M$: SRAM size in values, with $d\le M\le Nd$
- $N^2d^2/M$: $Q$ and $O$ are streamed once per key block: $T_c=N/B_c\approx 4Nd/M$ times $Nd$ values. Since $d^2\ll M$ (e.g. $64^2=4096$ against about $10^5$), this is many times smaller than $N^2$
- $O(N)$: beyond inputs and outputs, only $m$ and $\ell$ (FlashAttention-2: $L$) per row are kept for the backward pass

The paper also proves a lower bound: no exact attention algorithm can asymptotically use fewer HBM accesses than $\Theta(N^2d^2/M)$ for all $M$ in $[d,Nd]$. FLOPs stay the same, $O(N^2d)$: FlashAttention computes the same matrix products, only in a different order and without the round trips. With a causal mask, tiles that lie entirely above the diagonal are skipped, which saves about half the work (see **Causal attention**). **Block-sparse FlashAttention** skips every tile that a given block mask zeroes. Its IO cost is proportional to the fraction of non-zero blocks, and it was faster than all approximate methods the authors tested.

## Backward pass by recomputation

$$
P_{ij}=\exp\!\big(S_{ij}-L_i\big),\qquad dV=P^\top dO,\qquad dP=dO\,V^\top,\qquad D_i=\mathrm{rowsum}(dO_i\circ O_i),\qquad dS=P\circ(dP-D),\qquad dQ=\frac{dS\,K}{\sqrt d},\quad dK=\frac{dS^\top Q}{\sqrt d}
$$

- $L_i=m_i+\log\ell_i$: saved log-sum-exp of row $i$; with it, any tile of $P$ is recomputed exactly from $Q_i$ and $K_j$
- $D_i$: equals $\sum_j P_{ij}\,dP_{ij}$, the term the softmax Jacobian needs; computing it from $dO$ and $O$ avoids storing $P$
- $dS$: gradient of the scores, formed tile by tile and immediately contracted into $dQ$ and $dK$

Standard attention saves $P$ ($N^2$ values per head) for the backward pass. FlashAttention saves only $O$ and $L$. In the backward pass it reloads $Q$, $K$, $V$, $O$ and $dO$ block by block and recomputes each tile of $S$ and $P$. Dropout is handled the same way: the kernel saves the random generator state and regenerates the same mask instead of storing it. This is gradient checkpointing applied inside one kernel. The extra FLOPs are cheap because the recomputation happens in SRAM, and the saved HBM reads of $P$ were the expensive part.

GPT-2 medium attention on an A100, forward + backward (seq. length 1024, head size 64, 16 heads, batch 64; Dao et al. 2022, Fig. 2)

| metric | standard attention | FlashAttention |
| --- | --- | --- |
| GFLOPs | 66.6 | 75.2 |
| HBM reads + writes | 40.3 GB | 4.4 GB |
| runtime | 41.7 ms | 7.3 ms |

**End-to-end results reported in the paper**

- **BERT-large:** sequence length 512: 15% faster end-to-end training than the MLPerf 1.1 training speed record
- **GPT-2:** sequence length 1K: up to 3× faster training than the HuggingFace implementation
- **Long-Range Arena:** sequence lengths 1K–4K: 2.4× speed-up
- **Longer context:** GPT-2 with 4K context trained faster than Megatron-LM at 1K and reached 0.7 better perplexity; 6.4 points of lift on long-document classification
- **Path-X / Path-256:** first Transformers to beat chance: 61.4% at sequence length 16K and, with block-sparse FlashAttention, 63.1% at 64K

> Figure: A 16×16 score matrix (d = 4, random Q, K, V) processed in 4×4 tiles in the order of Algorithm 1: for each K/V block, every Q block. Row 10 lies in Q block 2. After the tile with K block 0 its running max is m = 1.172 and ℓ = 2.732, and its output is still far from the exact one (max error 1.4). At tile 11, K block 2 contains a larger score: m jumps to 2.657, and the old ℓ and O are multiplied by e^(1.172 − 2.657) = 0.226. After tile 15 the row matches exact softmax attention to 1.1e-16, with m = 2.657 and ℓ = 3.204 identical to the two-pass values. The toy counters count HBM values read and written. With 4×4 tiles FlashAttention moves 1152 values, against 1280 for standard attention, which writes and rereads S and P. With 2×2 tiles it moves 2176, with 8×8 only 640: larger tiles mean fewer reloads of Q and O. The lower right applies the same IO model at full size, with an SRAM of 64K values. At N = 4096 and d = 64 it gives 130.0 MiB against 25.5 MiB per head (5.10× fewer accesses), and at N = 16384 it gives 2.01 GiB against 396 MiB. The ratio is about M/3d², so it drops to 1.33× at d = 128. The real gain is larger, because standard implementations also make extra N² passes for masking and dropout. For the backward pass, standard attention keeps P (32 MiB at N = 4096); FlashAttention keeps 16 KiB of row statistics.

## Worked example: one query, two key blocks

One query row has scores $s=(1,\,2\,|\,3,\,0)$ against four keys, split into two blocks of two. The values are $v_1=(1,0)$, $v_2=(0,1)$, $v_3=(2,2)$, $v_4=(-1,1)$.

Online-softmax state after each block (computed in Node); Õ is the unnormalised output

| after | m | ℓ | rescale e^(m_old − m) | Õ | O = Õ/ℓ |
| --- | --- | --- | --- | --- | --- |
| start | −∞ | 0 |  | (0, 0) |  |
| block 1 (s = 1, 2) | 2 | e⁻¹ + 1 = 1.3679 | 0 | (0.3679, 1.0000) | (0.2689, 0.7311) |
| block 2 (s = 3, 0) | 3 | 0.3679·1.3679 + 1 + e⁻³ = 1.5530 | e⁻¹ = 0.3679 | (2.0855, 2.4177) | (1.3429, 1.5568) |
| exact, two passes | 3 | 1.5530 |  |  | (1.3429, 1.5568) |

**Step by step**

1. Block 1: tile maximum 2, $p=(e^{-1},e^{0})=(0.3679,1)$, $\ell=1.3679$, $\tilde O=0.3679\,v_1+1\cdot v_2=(0.3679,\,1)$. If the row ended here, $O=(0.2689,\,0.7311)$.
2. Block 2: tile maximum 3 exceeds 2, so $m=3$ and the old state is multiplied by $e^{2-3}=0.3679$. New terms $e^{3-3}=1$ and $e^{0-3}=0.0498$.
3. $\ell=0.3679\cdot1.3679+1+0.0498=1.5530$ and $\tilde O=0.3679\,(0.3679,1)+1\cdot(2,2)+0.0498\,(-1,1)=(2.0855,\,2.4177)$.
4. $O=\tilde O/\ell=(1.3429,\,1.5568)$, identical to $\mathrm{softmax}(1,2,3,0)=(0.0871,\,0.2369,\,0.6439,\,0.0321)$ applied to $V$. The saved statistic is $L=3+\log1.5530=3.4402$, from which the backward pass recomputes $P_{1j}=e^{s_j-3.4402}$.

```python
import torch, torch.nn.functional as F

def flash_forward(q, k, v, Br=64, Bc=64):
    # q, k, v: (N, d) for one head. Returns O (N, d) and log-sum-exp L (N,)
    N, d = q.shape
    O, L = torch.empty_like(q), torch.empty(N)
    for i in range(0, N, Br):                        # one "thread block" per query block
        qi = q[i:i + Br] * d ** -0.5                 # (Br, d) kept in SRAM
        m = torch.full((qi.shape[0],), float("-inf"))
        l = torch.zeros(qi.shape[0])
        acc = torch.zeros(qi.shape[0], d)            # unnormalised output
        for j in range(0, N, Bc):
            s = qi @ k[j:j + Bc].T                   # (Br, Bc) tile, never stored in full
            m_new = torch.maximum(m, s.max(-1).values)
            p = torch.exp(s - m_new[:, None])
            alpha = torch.exp(m - m_new)             # rescales the old statistics (0 at first)
            l = alpha * l + p.sum(-1)
            acc = alpha[:, None] * acc + p @ v[j:j + Bc]
            m = m_new
        O[i:i + Br] = acc / l[:, None]               # normalise once at the end
        L[i:i + Br] = m + torch.log(l)               # all the backward pass needs
    return O, L

torch.manual_seed(0)
N, d = 1000, 64                                      # N need not be a multiple of the block size
q, k, v = (torch.randn(N, d) for _ in range(3))
O, L = flash_forward(q, k, v)
ref = F.scaled_dot_product_attention(q[None], k[None], v[None])[0]
print(O.shape, L.shape)                              # torch.Size([1000, 64]) torch.Size([1000])
print(torch.allclose(O, ref, atol=1e-5))             # True

# the backward pass rebuilds P from L instead of storing it
P = torch.softmax(q @ k.T / d ** 0.5, dim=-1)        # (1000, 1000)
print(torch.allclose(P, torch.exp(q @ k.T / d ** 0.5 - L[:, None]), atol=1e-6))  # True
```

Reference tiled forward pass with online softmax (FlashAttention-2 loop order), checked against PyTorch SDPA

This loop shows the arithmetic, not the speed: in Python every tile is still a separate tensor in GPU memory. The speed comes from running both loops inside one CUDA kernel. In PyTorch 2.2 and later, `F.scaled_dot_product_attention` dispatches to a FlashAttention-2 kernel when the GPU, dtype (fp16/bf16) and head size allow it, and `torch.nn.attention.sdpa_kernel(SDPBackend.FLASH_ATTENTION)` forces that backend. The `flash-attn` package provides the authors' kernels, including variable-length batches and sliding windows.

## FlashAttention-2 and FlashAttention-3

**FlashAttention-2** (Tri Dao, 2023; ICLR 2024) keeps the algorithm and reorganises the work. It reduces non-matmul FLOPs: the output stays unnormalised until the end, and only $L$ is saved instead of $m$ and $\ell$. It swaps the loops so the outer loop runs over query blocks. Each query block is then an independent thread block, and the kernel parallelises over sequence length as well as batch and heads, which helps when batch × heads is small. Inside a thread block it splits $Q$ rather than $K$ across warps, which removes communication through shared memory. It is about 2× faster than FlashAttention, reaching 50–73% of the A100's theoretical peak FLOPs/s and up to 225 TFLOPs/s per A100 when training GPT-style models (72% model FLOPs utilisation).

**FlashAttention-3** (Shah, Bikshandi, Zhang, Thakkar, Ramani and Dao, 2024; NeurIPS 2024) targets the Hopper H100, where FlashAttention-2 used only about 35% of peak. It uses the asynchronous Tensor Cores and the Tensor Memory Accelerator through **warp specialisation**: producer warps load tiles while consumer warps compute. "Ping-pong" scheduling between warpgroups, plus pipelining within one warpgroup, overlaps the softmax of one tile with the matrix products of another. It adds FP8 support with block quantisation and **incoherent processing**, a random orthogonal (Hadamard) transform that spreads outliers before quantising. In FP16 it is 1.5–2.0× faster than FlashAttention-2, up to 740 TFLOPs/s (75% utilisation). In FP8 it approaches 1.2 PFLOPs/s, with 2.6× lower numerical error than a baseline FP8 attention.

> **More FLOPs, less time:** FlashAttention does more arithmetic than standard attention (75.2 against 66.6 GFLOPs in the paper's GPT-2 example) and runs 5.7× faster, because it moves 9× less data through HBM. For memory-bound operations, the cost model to use counts bytes moved, not FLOPs. The same reasoning motivates kernel fusion, recomputation and the KV-cache optimisations used in inference.

> **It is exact, but not bit-identical:** FlashAttention sums in a different order, so its outputs differ from a reference implementation at the level of rounding error (larger in fp16/bf16). Compare with a tolerance. Runs can also be nondeterministic: FlashAttention-2's backward pass accumulates $dQ$ with atomic adds by default. If you need bitwise reproducibility, use the deterministic option, which is slower.

> **What it does not change:** FlashAttention still computes all $N^2$ scores (unless a block-sparse or causal mask lets it skip tiles), so time remains quadratic in $N$. What becomes linear is memory, which is why it made long contexts trainable. It also changes nothing about the model: the same weights give the same function. During decoding, with one query per step, the bottleneck is reading the KV cache. Variants such as Flash-Decoding split that work along the key sequence.

**Strengths**

- Exact attention; drop-in replacement with the same outputs up to rounding
- Extra memory $O(N)$ instead of $O(N^2)$, which enables much longer sequences
- Several times less HBM traffic and faster wall-clock attention (5.7× in the paper's GPT-2 example)
- Masking, dropout and causal or block-sparse tile skipping fused in the same kernel

**Limitations**

- Still $O(N^2d)$ FLOPs; long contexts remain quadratic in time
- Hand-written, hardware-specific kernels; new GPUs needed a redesign (FlashAttention-3 for Hopper)
- Restrictions on dtype and head size; arbitrary attention biases or custom score functions need kernel support
- Recomputation adds backward FLOPs, and the default backward pass is nondeterministic

**Question:** While processing key block j, a query row's running maximum rises from m = 2 to m = 3. What must FlashAttention do with its accumulated ℓ and Õ?

- A. Nothing; the new block is simply added
- B. Multiply both by e^(2 − 3) = e⁻¹ before adding the new block's terms, which are computed relative to m = 3
- C. Recompute all previous blocks from scratch with m = 3
- D. Divide both by 3

**Answer:** B. Multiply both by e^(2 − 3) = e⁻¹ before adding the new block's terms, which are computed relative to m = 3 — Previous terms were $e^{s-2}$; relative to the new maximum they must be $e^{s-3}=e^{s-2}e^{-1}$. One multiplication by $e^{m_{\text{old}}-m_{\text{new}}}$ fixes both the normaliser and the weighted value sum, so no block is revisited.

**Question:** Which quantity does FlashAttention store for the backward pass instead of the attention matrix P?

- A. Nothing; it uses a closed-form gradient
- B. The score matrix S in fp8
- C. The output O and one log-sum-exp (or m and ℓ) per query row; tiles of P are recomputed from Q, K and these statistics
- D. A low-rank approximation of P

**Answer:** C. The output O and one log-sum-exp (or m and ℓ) per query row; tiles of P are recomputed from Q, K and these statistics — $P_{ij}=\exp(q_i^\top k_j/\sqrt d-L_i)$, so $L$ plus the inputs determine $P$ exactly. $D_i=\mathrm{rowsum}(dO_i\circ O_i)$ supplies the remaining softmax-gradient term. Storage is $O(N)$ instead of $O(N^2)$.

**Key points**

- Online softmax: $m'=\max(m,\mathrm{rowmax}\,S_j)$, $\ell'=e^{m-m'}\ell+\mathrm{rowsum}\,e^{S_j-m'}$, $\tilde O'=e^{m-m'}\tilde O+e^{S_j-m'}V_j$; at the end $O=\tilde O/\ell$ is exact.
- IO complexity: standard attention $\Theta(Nd+N^2)$ HBM accesses, FlashAttention $\Theta(N^2d^2/M)$ for SRAM size $M$; extra memory $O(N)$ instead of $O(N^2)$.
- The N×N matrices S and P stay in SRAM tile by tile; the backward pass recomputes them from Q, K, V and the saved log-sum-exp.
- More FLOPs, much less memory traffic: in the paper's GPT-2 example 4.4 GB instead of 40.3 GB of HBM traffic and 7.3 ms instead of 41.7 ms.
- FlashAttention-2 (2023): loops swapped, parallel over sequence length, about 2× faster. FlashAttention-3 (2024): Hopper asynchrony, warp specialisation and FP8, up to 740 TFLOPs/s in FP16.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
