Self-attention

Every token of a sequence is projected into a query, a key and a value, and each output is a softmax-weighted average of all the values, with weights from query–key dot products. Any token can reach any other in one step, the layer is permutation-equivariant, and it costs O(n²) in sequence length.

14 min read1083 wordscore0 viewsupdated 2026-10-12

Definition

Self-attention (also intra-attention) is attention in which the queries, keys and values all come from the same sequence. Given token representations X∈Rn×dmodelX\in\mathbb R^{n\times d_{\text{model}}}, a self-attention layer forms Q=XWQQ=XW_Q, K=XWKK=XW_K, V=XWVV=XW_V and returns Y=softmax(QK⊤/dk) VY=\mathrm{softmax}(QK^\top/\sqrt{d_k})\,V. Row ii of YY is a mixture of the values of all nn tokens, weighted by how well the query of token ii matches each token's key. It is the core layer of every Transformer.

In the RNN translation models of 2014–2015 (see Attention mechanism), attention connected two different sequences: decoder states queried encoder states. A recurrent network still built the representation of each sentence, one token after another. The idea of letting a sequence attend to itself appeared in 2016. Cheng, Dong and Lapata ("Long Short-Term Memory-Networks for Machine Reading", EMNLP 2016) added intra-attention over an LSTM's own past states. Parikh et al. (EMNLP 2016) used intra-sentence attention in a decomposable model for natural language inference, and Lin et al. (ICLR 2017) built sentence embeddings from self-attention over BiLSTM states.

Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), removed the recurrence entirely. Their Transformer builds every representation from stacked self-attention and position-wise feed-forward layers, with Positional encoding added to the inputs. On WMT 2014 English→German the big model reached 28.4 BLEU, more than 2 BLEU above the best previous results including ensembles, after 3.5 days of training on 8 P100 GPUs. The argument of the paper was less about accuracy than about structure: self-attention connects every pair of positions with a constant number of operations, and all positions are computed in parallel.

The layer

Q=XWQ,K=XWK,V=XWV,A=softmax ⁣(QK⊤dk)∈Rn×n,Y=AV,yi=∑j=1nAij vjQ=XW_Q,\quad K=XW_K,\quad V=XW_V,\qquad A=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)\in\mathbb R^{n\times n},\qquad Y=AV,\qquad y_i=\sum_{j=1}^{n}A_{ij}\,v_j
X∈Rn×dmodelX\in\mathbb R^{n\times d_{\text{model}}}
one row per token: embeddings plus position information in the first layer, the previous layer's output later
WQ,WK∈Rdmodel×dkW_Q,W_K\in\mathbb R^{d_{\text{model}}\times d_k}
learned query and key projections; qi⊤kjq_i^\top k_j measures how relevant token jj is to token ii
WV∈Rdmodel×dvW_V\in\mathbb R^{d_{\text{model}}\times d_v}
learned value projection: what token jj contributes if it is attended to
AA
attention matrix; row ii is the distribution of token ii over all nn tokens (including itself), each row sums to 1
dk\sqrt{d_k}
keeps the scores at unit variance; derived in Scaled dot-product attention

The three projections are what make the layer useful. Without them (Q=K=V=XQ=K=V=X) the score xi⊤xjx_i^\top x_j is symmetric and usually largest for j=ij=i, so each token mostly averages itself with tokens that look like it. With separate WQW_Q and WKW_K, the relation "ii attends to jj" need not be symmetric: a pronoun's query can look for features that only nouns carry in their keys, while the noun's own query looks for something else. WVW_V then decides what information is copied along that edge. In practice the layer runs hh such heads in parallel with dk=dmodel/hd_k=d_{\text{model}}/h (see Multi-head attention).

Permutation equivariance

Self-attention has no notion of order. Let PP be a permutation matrix that reorders the tokens. Then (PX)WQ=PQ(PX)W_Q=PQ, the same for KK and VV, and the score matrix becomes PQK⊤P⊤PQK^\top P^\top. A row-wise softmax commutes with permuting rows and columns, so:

SA(PX)=softmax ⁣(PQK⊤P⊤dk)PV=P softmax ⁣(QK⊤dk)P⊤P V=P SA(X)\mathrm{SA}(PX)=\mathrm{softmax}\!\Big(\frac{PQK^\top P^\top}{\sqrt{d_k}}\Big)PV=P\,\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)P^\top P\,V=P\,\mathrm{SA}(X)
P⊤P=IP^\top P=I
permutation matrices are orthogonal
SA(PX)=P SA(X)\mathrm{SA}(PX)=P\,\mathrm{SA}(X)
shuffling the input tokens shuffles the outputs in the same way and changes nothing else: the layer sees a set, not a sequence

Two identical tokens at different positions therefore get identical outputs. In "the cat sat on the mat", both copies of "the" produce the same row of AA, so neither can tell which noun is its own. This is why Transformers add position information: absolute Positional encoding added to XX, relative schemes such as Rotary positional embeddings (RoPE) or ALiBi inside the scores, or a Causal attention mask, which also breaks the symmetry. For genuinely set-structured data (point clouds, object slots, the latent array of a Perceiver) the symmetry is exactly what you want.

Cost: constant path length, quadratic size

layer typecomplexity per layersequential operationsmaximum path length
self-attentionO(n2⋅d)O(n^2\cdot d)O(1)O(1)O(1)O(1)
recurrentO(n⋅d2)O(n\cdot d^2)O(n)O(n)O(n)O(n)
convolutional (kernel k)O(k⋅n⋅d2)O(k\cdot n\cdot d^2)O(1)O(1)O(log⁡kn)O(\log_k n)
self-attention restricted to r neighboursO(r⋅n⋅d)O(r\cdot n\cdot d)O(1)O(1)O(n/r)O(n/r)
Per-layer cost for sequence length n and width d (Vaswani et al. 2017, Table 1)

Any two tokens interact through one attention step, so the gradient path between them has constant length. An RNN needs O(n)O(n) steps, which is where vanishing gradients and slow sequential training come from. The price is the n×nn\times n matrix. Computing QK⊤QK^\top and AVAV takes 4n2d4n^2d floating-point operations per layer (counting multiply and add), against roughly 24nd224nd^2 for the projections and the 4d4d feed-forward block. Attention is the cheaper part while n<6dn<6d; Vaswani et al. note that self-attention is faster than recurrence when nn is smaller than dd, which was the case for sentence-level translation. Memory is the harder limit. Stored naively, the weights take n2n^2 numbers per head and sequence. FlashAttention avoids storing them, and Sparse attention, Linear attention, Longformer and Linformer change the n2n^2 itself.

Q = XW_Q, K = XW_K, V = XW_V from the same X · A = softmax(QKᵀ/√dₖ) · Y = AVtheanimaldidn'tcrossthestreetbecauseitwastiredtheanimaldidn'tcrossthestreetbecauseitwastiredrows = queries, cols = keys, same tokens · click a rowquery “it” at position 7, top keys:animal 0.673tired 0.066it 0.048vectors of “it” (dₖ = 4)qᵢkᵢvᵢyᵢorder original · positions nonemax |Y(PX) − P·Y(X)| = 0.0e+0permutation equivariantthe two “the” rows: identical“it” → animal 0.673, street 0.017one layer at n = 4,096 tokens, d_model = 768, 12 heads, dₖ = 64:scores per head n² = 16.8M · fp16 attention matrix per head 33.6 MBQKᵀ + AV: 4n²d = 51.5GFLOP · projections + FFN: 24nd² = 58.0GFLOPattention share 47.1%doubling n: dense FLOPs ×2, attention FLOPs ×4; they are equal at n = 6d = 4608
projections
positions
fig.One attention head over the 10 tokens of 'the animal didn't cross the street because it was tired'. Embeddings are hand-set word features plus noise, and W_Q, W_K encode four relations: pronouns and adjectives look for animate nouns, verbs and determiners look for nouns, auxiliaries look for verbs. With the projections, 'it' puts 0.673 on 'animal' and 0.017 on 'street', 'tired' puts 0.514 on 'animal', and 'cross' puts 0.354 on 'street'. Both copies of 'the' get the same row (0.336 on 'street', 0.170 on 'animal'): without positions, neither can know which noun follows it. Switch to Q = K = V = X: the weights flatten and 8 of 10 rows put their largest weight on the token itself (the two 'the' rows split it between the two copies). Shuffle the tokens: the matrix is permuted, and the outputs are the permuted outputs, max |Y(PX) − P·Y(X)| about 1e−16. Add sinusoidal positions: the two 'the' rows now differ, 'it' → 'animal' rises to 0.831, and a shuffle changes the outputs by up to 0.302. The bottom strip counts the cost of one BERT-base-sized layer (d_model = 768): at n = 512 attention is 10.0% of the FLOPs; at n = 4096 it is 47.1% and each head's fp16 weight matrix takes 33.6 MB; at n = 131,072 it is 96.6%.

Worked example: three tokens, one head

Take three tokens with dmodel=dk=2d_{\text{model}}=d_k=2: x1=(1,0)x_1=(1,0), x2=(0,2)x_2=(0,2), x3=(1,1)x_3=(1,1). Let WQ=IW_Q=I, WK=(0110)W_K=\begin{pmatrix}0&1\\1&0\end{pmatrix} (it swaps the two coordinates) and WV=(1201)W_V=\begin{pmatrix}1&2\\0&1\end{pmatrix}. Then Q=XQ=X, k1=(0,1)k_1=(0,1), k2=(2,0)k_2=(2,0), k3=(1,1)k_3=(1,1), and v1=(1,2)v_1=(1,2), v2=(0,2)v_2=(0,2), v3=(1,3)v_3=(1,3).

querys·1s·2s·3A·1A·2A·3yᵢ = Σⱼ Aᵢⱼvⱼ
x₁ = (1, 0)0.00001.41420.70710.14000.57600.2840(0.4240, 2.2840)
x₂ = (0, 2)1.41420.00001.41420.44580.10840.4458(0.8916, 2.4458)
x₃ = (1, 1)0.70711.41421.41420.19780.40110.4011(0.5989, 2.4011)
Scores qᵢᵀkⱼ/√2 and weights (rows: queries; checked in PyTorch)

Reading the table

  1. 1Row 1: q1⊤k2=(1,0)⋅(2,0)=2q_1^\top k_2=(1,0)\cdot(2,0)=2, divided by 2\sqrt2 gives 1.41421.4142. Token 1 attends most to token 2 (0.576), although x1x_1 and x2x_2 are orthogonal: WKW_K decides what a match means.
  2. 2AA is not symmetric: A12=0.576A_{12}=0.576 but A21=0.108A_{21}=0.108. Each row is a distribution of its own; columns need not sum to 1.
  3. 3Output: y1=0.14 (1,2)+0.576 (0,2)+0.284 (1,3)=(0.4240, 2.2840)y_1=0.14\,(1,2)+0.576\,(0,2)+0.284\,(1,3)=(0.4240,\,2.2840).
  4. 4Without projections (Q=K=V=XQ=K=V=X) the scores are xi⊤xj/2x_i^\top x_j/\sqrt2 and token 2 gives 0.7679 to itself, because ∥x2∥2=4\|x_2\|^2=4 is the largest dot product in its row.
  5. 5Permute the tokens to the order (3,1,2)(3,1,2): the outputs come out as (y3,y1,y2)(y_3,y_1,y_2) exactly, as the equivariance identity says.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class SelfAttention(nn.Module):
4 """Single-head self-attention: Q, K and V are three projections of the same X."""
5 def __init__(self, d_model, d_k):
6 super().__init__()
7 self.q = nn.Linear(d_model, d_k, bias=False)
8 self.k = nn.Linear(d_model, d_k, bias=False)
9 self.v = nn.Linear(d_model, d_k, bias=False)
10
11 def forward(self, x, return_weights=False):
12 # x (N, n, d_model)
13 q, k, v = self.q(x), self.k(x), self.v(x) # each (N, n, d_k)
14 scores = q @ k.transpose(-2, -1) / k.size(-1) ** 0.5 # (N, n, n)
15 a = scores.softmax(dim=-1) # each row sums to 1
16 y = a @ v # (N, n, d_k)
17 return (y, a) if return_weights else y
18
19torch.manual_seed(0)
20N, n, d_model, d_k = 2, 10, 32, 16
21x = torch.randn(N, n, d_model)
22sa = SelfAttention(d_model, d_k)
23y, a = sa(x, return_weights=True)
24print(y.shape, a.shape) # torch.Size([2, 10, 16]) torch.Size([2, 10, 10])
25print(torch.allclose(a.sum(-1), torch.ones(N, n))) # True
26
27# Permutation equivariance: shuffling the tokens shuffles the outputs the same way.
28perm = torch.randperm(n)
29print(torch.allclose(sa(x[:, perm]), y[:, perm], atol=1e-6)) # True
30
31# Same result from PyTorch's fused kernel.
32y2 = F.scaled_dot_product_attention(sa.q(x), sa.k(x), sa.v(x))
33print(torch.allclose(y, y2, atol=1e-6)) # True
34
35# Adding a position vector per slot breaks the symmetry.
36pos = torch.randn(n, d_model)
37print(torch.allclose(sa(x[:, perm] + pos), sa(x + pos)[:, perm], atol=1e-6)) # False
38
39# The n x n weight matrix grows quadratically: fp16 bytes per head, per sequence.
40for m in (1024, 8192, 65536):
41 print(m, f"{m * m * 2 / 2**20:.0f} MiB") # 2 MiB, 128 MiB, 8192 MiB
Single-head self-attention from scratch, checked against PyTorch's fused kernel and for permutation equivariance

Self-attention in the original Transformer (Vaswani et al. 2017)

Where
encoder: one self-attention sub-layer per layer over the source; decoder: masked self-attention over the target prefix, then Cross-attention to the encoder
Base model
N=6N=6 layers, dmodel=512d_{\text{model}}=512, h=8h=8 heads, dk=dv=64d_k=d_v=64, feed-forward dff=2048d_{ff}=2048
Big model
dmodel=1024d_{\text{model}}=1024, h=16h=16, dff=4096d_{ff}=4096
Each sub-layer
LayerNorm(x+Sublayer(x))\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x)) with dropout 0.1 on the sub-layer output (post-LN)
WMT'14 En→De
27.3 BLEU (base, 12 hours on 8 P100), 28.4 BLEU (big, 3.5 days)
WMT'14 En→Fr
41.8 BLEU (big)

Use when

  • +Constant path length between any two tokens: long-range dependencies are one step away
  • +All positions computed in parallel with a few large matrix products, unlike an RNN
  • +Input-dependent mixing weights; works for any sequence length with the same parameters
  • +Permutation equivariance is a clean inductive bias for sets

Avoid when

  • −O(n2d)O(n^2d) time and O(n2)O(n^2) memory per head for the attention weights
  • −No built-in notion of order: needs positional information
  • −No locality prior, so it typically needs more data than CNNs or RNNs on small datasets
  • −Attention weights invite over-interpretation as explanations

Check yourself

You feed a Transformer encoder layer (no positional encoding) the tokens [A, B, C] and then [C, A, B]. How are the two outputs related?

Check yourself

A model has d_model = 1024. Using the 4n²d versus 24nd² count, from about which sequence length does attention (QKᵀ and AV) cost more FLOPs than the projections and feed-forward block of a layer?

Remember

  • ✓Q=XWQ, K=XWK, V=XWVQ=XW_Q,\ K=XW_K,\ V=XW_V from the same XX; Y=softmax(QK⊤/dk) VY=\mathrm{softmax}(QK^\top/\sqrt{d_k})\,V with A∈Rn×nA\in\mathbb R^{n\times n}, rows summing to 1.
  • ✓Permutation-equivariant: SA(PX)=P SA(X)\mathrm{SA}(PX)=P\,\mathrm{SA}(X), so positions must be added (positional encodings, RoPE, ALiBi, causal masks).
  • ✓Cost per layer O(n2d)O(n^2d) time and O(n2)O(n^2) memory per head, but O(1)O(1) sequential steps and O(1)O(1) path length between any two tokens.
  • ✓Separate WQW_Q and WKW_K make attention asymmetric; without them a token mostly attends to itself.
  • ✓Vaswani et al. (NeurIPS 2017) built the Transformer from self-attention alone: 28.4 BLEU on WMT'14 En→De.
← back to Deep Learning roadmap