Definition
Self-attention (also intra-attention) is attention in which the queries, keys and values all come from the same sequence. Given token representations , a self-attention layer forms , , and returns . Row of is a mixture of the values of all tokens, weighted by how well the query of token matches each token's key. It is the core layer of every Transformer.
In the RNN translation models of 2014–2015 (see Attention mechanism), attention connected two different sequences: decoder states queried encoder states. A recurrent network still built the representation of each sentence, one token after another. The idea of letting a sequence attend to itself appeared in 2016. Cheng, Dong and Lapata ("Long Short-Term Memory-Networks for Machine Reading", EMNLP 2016) added intra-attention over an LSTM's own past states. Parikh et al. (EMNLP 2016) used intra-sentence attention in a decomposable model for natural language inference, and Lin et al. (ICLR 2017) built sentence embeddings from self-attention over BiLSTM states.
Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), removed the recurrence entirely. Their Transformer builds every representation from stacked self-attention and position-wise feed-forward layers, with Positional encoding added to the inputs. On WMT 2014 English→German the big model reached 28.4 BLEU, more than 2 BLEU above the best previous results including ensembles, after 3.5 days of training on 8 P100 GPUs. The argument of the paper was less about accuracy than about structure: self-attention connects every pair of positions with a constant number of operations, and all positions are computed in parallel.
The layer
- one row per token: embeddings plus position information in the first layer, the previous layer's output later
- learned query and key projections; measures how relevant token is to token
- learned value projection: what token contributes if it is attended to
- attention matrix; row is the distribution of token over all tokens (including itself), each row sums to 1
- keeps the scores at unit variance; derived in Scaled dot-product attention
The three projections are what make the layer useful. Without them () the score is symmetric and usually largest for , so each token mostly averages itself with tokens that look like it. With separate and , the relation " attends to " need not be symmetric: a pronoun's query can look for features that only nouns carry in their keys, while the noun's own query looks for something else. then decides what information is copied along that edge. In practice the layer runs such heads in parallel with (see Multi-head attention).
Permutation equivariance
Self-attention has no notion of order. Let be a permutation matrix that reorders the tokens. Then , the same for and , and the score matrix becomes . A row-wise softmax commutes with permuting rows and columns, so:
- permutation matrices are orthogonal
- shuffling the input tokens shuffles the outputs in the same way and changes nothing else: the layer sees a set, not a sequence
Two identical tokens at different positions therefore get identical outputs. In "the cat sat on the mat", both copies of "the" produce the same row of , so neither can tell which noun is its own. This is why Transformers add position information: absolute Positional encoding added to , relative schemes such as Rotary positional embeddings (RoPE) or ALiBi inside the scores, or a Causal attention mask, which also breaks the symmetry. For genuinely set-structured data (point clouds, object slots, the latent array of a Perceiver) the symmetry is exactly what you want.
Cost: constant path length, quadratic size
| layer type | complexity per layer | sequential operations | maximum path length |
|---|---|---|---|
| self-attention | |||
| recurrent | |||
| convolutional (kernel k) | |||
| self-attention restricted to r neighbours |
Any two tokens interact through one attention step, so the gradient path between them has constant length. An RNN needs steps, which is where vanishing gradients and slow sequential training come from. The price is the matrix. Computing and takes floating-point operations per layer (counting multiply and add), against roughly for the projections and the feed-forward block. Attention is the cheaper part while ; Vaswani et al. note that self-attention is faster than recurrence when is smaller than , which was the case for sentence-level translation. Memory is the harder limit. Stored naively, the weights take numbers per head and sequence. FlashAttention avoids storing them, and Sparse attention, Linear attention, Longformer and Linformer change the itself.
Worked example: three tokens, one head
Take three tokens with : , , . Let , (it swaps the two coordinates) and . Then , , , , and , , .
| query | s·1 | s·2 | s·3 | A·1 | A·2 | A·3 | yᵢ = Σⱼ Aᵢⱼvⱼ |
|---|---|---|---|---|---|---|---|
| x₁ = (1, 0) | 0.0000 | 1.4142 | 0.7071 | 0.1400 | 0.5760 | 0.2840 | (0.4240, 2.2840) |
| x₂ = (0, 2) | 1.4142 | 0.0000 | 1.4142 | 0.4458 | 0.1084 | 0.4458 | (0.8916, 2.4458) |
| x₃ = (1, 1) | 0.7071 | 1.4142 | 1.4142 | 0.1978 | 0.4011 | 0.4011 | (0.5989, 2.4011) |
Reading the table
- 1Row 1: , divided by gives . Token 1 attends most to token 2 (0.576), although and are orthogonal: decides what a match means.
- 2 is not symmetric: but . Each row is a distribution of its own; columns need not sum to 1.
- 3Output: .
- 4Without projections () the scores are and token 2 gives 0.7679 to itself, because is the largest dot product in its row.
- 5Permute the tokens to the order : the outputs come out as exactly, as the equivariance identity says.
1import torch, torch.nn as nn, torch.nn.functional as F23class SelfAttention(nn.Module):4 """Single-head self-attention: Q, K and V are three projections of the same X."""5 def __init__(self, d_model, d_k):6 super().__init__()7 self.q = nn.Linear(d_model, d_k, bias=False)8 self.k = nn.Linear(d_model, d_k, bias=False)9 self.v = nn.Linear(d_model, d_k, bias=False)1011 def forward(self, x, return_weights=False):12 # x (N, n, d_model)13 q, k, v = self.q(x), self.k(x), self.v(x) # each (N, n, d_k)14 scores = q @ k.transpose(-2, -1) / k.size(-1) ** 0.5 # (N, n, n)15 a = scores.softmax(dim=-1) # each row sums to 116 y = a @ v # (N, n, d_k)17 return (y, a) if return_weights else y1819torch.manual_seed(0)20N, n, d_model, d_k = 2, 10, 32, 1621x = torch.randn(N, n, d_model)22sa = SelfAttention(d_model, d_k)23y, a = sa(x, return_weights=True)24print(y.shape, a.shape) # torch.Size([2, 10, 16]) torch.Size([2, 10, 10])25print(torch.allclose(a.sum(-1), torch.ones(N, n))) # True2627# Permutation equivariance: shuffling the tokens shuffles the outputs the same way.28perm = torch.randperm(n)29print(torch.allclose(sa(x[:, perm]), y[:, perm], atol=1e-6)) # True3031# Same result from PyTorch's fused kernel.32y2 = F.scaled_dot_product_attention(sa.q(x), sa.k(x), sa.v(x))33print(torch.allclose(y, y2, atol=1e-6)) # True3435# Adding a position vector per slot breaks the symmetry.36pos = torch.randn(n, d_model)37print(torch.allclose(sa(x[:, perm] + pos), sa(x + pos)[:, perm], atol=1e-6)) # False3839# The n x n weight matrix grows quadratically: fp16 bytes per head, per sequence.40for m in (1024, 8192, 65536):41 print(m, f"{m * m * 2 / 2**20:.0f} MiB") # 2 MiB, 128 MiB, 8192 MiBSelf-attention in the original Transformer (Vaswani et al. 2017)
- Where
- encoder: one self-attention sub-layer per layer over the source; decoder: masked self-attention over the target prefix, then Cross-attention to the encoder
- Base model
- layers, , heads, , feed-forward
- Big model
- , ,
- Each sub-layer
- with dropout 0.1 on the sub-layer output (post-LN)
- WMT'14 En→De
- 27.3 BLEU (base, 12 hours on 8 P100), 28.4 BLEU (big, 3.5 days)
- WMT'14 En→Fr
- 41.8 BLEU (big)
Use when
- +Constant path length between any two tokens: long-range dependencies are one step away
- +All positions computed in parallel with a few large matrix products, unlike an RNN
- +Input-dependent mixing weights; works for any sequence length with the same parameters
- +Permutation equivariance is a clean inductive bias for sets
Avoid when
- − time and memory per head for the attention weights
- −No built-in notion of order: needs positional information
- −No locality prior, so it typically needs more data than CNNs or RNNs on small datasets
- −Attention weights invite over-interpretation as explanations
Check yourself
You feed a Transformer encoder layer (no positional encoding) the tokens [A, B, C] and then [C, A, B]. How are the two outputs related?
Check yourself
A model has d_model = 1024. Using the 4n²d versus 24nd² count, from about which sequence length does attention (QKᵀ and AV) cost more FLOPs than the projections and feed-forward block of a layer?
Remember
- ✓ from the same ; with , rows summing to 1.
- ✓Permutation-equivariant: , so positions must be added (positional encodings, RoPE, ALiBi, causal masks).
- ✓Cost per layer time and memory per head, but sequential steps and path length between any two tokens.
- ✓Separate and make attention asymmetric; without them a token mostly attends to itself.
- ✓Vaswani et al. (NeurIPS 2017) built the Transformer from self-attention alone: 28.4 BLEU on WMT'14 En→De.