Definition
Attention mechanism a layer that, given a query and a set of key–value pairs , computes a score for every key, normalises the scores with a softmax into attention weights , and returns the context vector . Every step is differentiable, so what to attend to is learned by backpropagation.
An Encoder–decoder architecture built from RNNs compresses the whole source into one fixed-size vector, and translation quality drops as sentences grow longer (Cho et al., 2014). Attention removes that bottleneck. The encoder keeps one vector per input position. At every output step the decoder forms a query from its current state and reads a fresh, input-dependent mixture of those vectors. Long inputs no longer have to fit into one vector, and gradients reach each input position through a short path instead of through the whole recurrence.
The ingredients appeared within a few years. Graves (2013) used a soft Gaussian window over the character string to let an RNN write handwriting for a given text. Bahdanau, Cho and Bengio (arXiv 2014, ICLR 2015) introduced the form used today for translation: an MLP scores each encoder annotation against the previous decoder state, and the softmax-weighted sum feeds the decoder. Graves, Wayne and Danihelka's Neural Turing Machine (2014) addressed an external memory by content (cosine similarity, sharpened by a key strength). Luong, Pham and Manning (2015) compared dot, general and concat scores and introduced local attention. Xu et al. (2015, "Show, Attend and Tell") separated soft from hard attention for image captioning. Vaswani et al. (2017) built the Transformer from scaled dot-product attention alone (see Self-Attention). Statistically, attention is a learned version of Nadaraya–Watson kernel regression (1964): predict at with a kernel-weighted average of stored outputs, .
Formula
- query: what the current step is looking for (in RNN translation, the decoder state)
- key of item : what the item is matched on (an encoder state, an image region, a memory slot)
- value of item : what is read out if the item is attended to; often in RNN models, separate projections in Transformers
- attention (alignment) weight: a probability distribution over the items
- temperature, usually 1; approaches a hard argmax, large a uniform average
- context vector: a convex combination of the values, so it lies inside their convex hull
With queries stacked as rows of , keys in and values in , the whole layer is two matrix products around a row-wise softmax: with . The matrix is the alignment matrix. Each row is one query's distribution over the items. Its cost is scores, which is why attention over long inputs is expensive.
Gradient
The softmax Jacobian gives a compact rule for how the context moves when one score changes:
- raising pulls the context toward , by an amount proportional to the weight already on it
- each value gets the upstream gradient scaled by its weight; items with learn nothing from this query
Two consequences follow. A score whose value equals the current context () has zero gradient. And once the softmax saturates (one ), every is close to zero, so the scores stop learning. This is the reason for the in scaled dot-product attention.
At a glance
- Inputs
- one query (or of them), keys and values
- Output
- a context vector per query, , plus the weights
- Parameters
- none in the mechanism itself; the score function (general , additive ) and the projections that produce are learned
- Cost
- time and memory for the weights
- Order
- permutation-invariant over the items: positions only matter through what is encoded in and
- Kinds
- cross-attention (queries from the decoder, keys/values from the encoder), self-attention (all three from the same sequence), soft vs hard, global vs local
The score function zoo
Everything except the score function is fixed. Four choices cover almost all RNN-era and Transformer models:
| name | score(q, k) | parameters | introduced in |
|---|---|---|---|
| dot | none (needs ) | Luong et al. 2015 | |
| scaled dot | none | Vaswani et al. 2017 | |
| general (bilinear, multiplicative) | Luong et al. 2015 | ||
| additive (concat, MLP) | with hidden size | Bahdanau et al. 2014 |
Dot says that a key matches when it points in the query's direction and is long. A long key wins even against a key that sits right next to the query. General inserts a learned matrix, so "match" can mean any linear relation between the two spaces, and it works across different sizes . Additive feeds both vectors through a one-hidden-layer MLP with tanh, which can represent non-bilinear similarity. It was the original choice and is still common when and come from different networks. For large , dot-product forms are much faster: the whole score matrix is one matrix multiply, while additive attention materialises an tensor of hidden units. Vaswani et al. note that the two perform similarly for small , and that additive beats unscaled dot for large . Their fix for dot is to scale it.
- the typical size of an unscaled score. Without the division, scores of size push the softmax toward a one-hot vector, where its gradients vanish
Worked example: three keys, scaled dot product
Query . Keys and values from the figure: cat , ; dog , ; car , . Here , so scores are divided by .
| key | qᵀk | s = qᵀk/√2 | exp(s) | a | a·v | ∂c/∂s = a(v − c) |
|---|---|---|---|---|---|---|
| cat | 1.40 | 0.9899 | 2.6911 | 0.3850 | 1.1551 | −0.1930 |
| dog | 1.80 | 1.2728 | 3.5708 | 0.5109 | 2.5543 | 0.7656 |
| car | −0.45 | −0.3182 | 0.7275 | 0.1041 | −0.2082 | −0.5726 |
| sum | 6.9894 | 1.0000 | 0.0000 |
Reading the table
- 1Scores: , , ; divide by .
- 2Softmax: , giving . Dog wins because its key is longest along .
- 3Context: .
- 4Gradient: raising dog's score moves toward 5 at rate ; raising car's moves it toward at rate . The three entries sum to 0 because adding the same constant to every score changes nothing.
Soft versus hard attention
Soft (deterministic) attention returns the expectation and is trained by ordinary backprop. Hard (stochastic) attention samples one index and returns . Xu et al. (2015) compared both for captioning, with over the locations of a VGG conv feature map. The sample is not differentiable, so they trained hard attention by maximising a variational lower bound on , which gives a REINFORCE-style estimator with a moving-average baseline and an entropy bonus. Mnih et al. (2014) had trained glimpse-based hard attention with REINFORCE the same way. Hard attention reads only one item per step, which is cheap at inference time, but its gradient estimate is noisy. Soft attention became the default; local attention (Luong et al., 2015) and sparse or top- variants sit in between.
Alignments you can look at
For a cross-attention layer, the alignment matrix (rows: output steps, columns: input positions) is a soft word alignment. Bahdanau et al. plotted it for English→French and found mostly diagonal matrices with off-diagonal blocks where French reorders adjectives and nouns. In captioning (Xu et al.), each word's weights over the 196 locations, upsampled onto the image, highlight the region the word describes. Plots like these are a cheap sanity check: a model that attends uniformly, or always to the final token, has not learned to use attention.
1import math, torch, torch.nn as nn, torch.nn.functional as F23class Attention(nn.Module):4 """query (N, Lq, dq), keys (N, Lk, dk), values (N, Lk, dv) -> context (N, Lq, dv), weights (N, Lq, Lk)"""5 def __init__(self, dq, dk, score="scaled", d_attn=64):6 super().__init__()7 self.score = score8 if score == "general":9 self.W = nn.Linear(dk, dq, bias=False) # q^T W k10 if score == "additive":11 self.Wq = nn.Linear(dq, d_attn, bias=False) # v^T tanh(W q + U k)12 self.Uk = nn.Linear(dk, d_attn, bias=False)13 self.v = nn.Linear(d_attn, 1, bias=False)1415 def forward(self, q, k, v, mask=None): # mask: (N, Lq, Lk) or (N, 1, Lk), True = keep16 if self.score == "dot":17 s = q @ k.transpose(-2, -1)18 elif self.score == "scaled":19 s = q @ k.transpose(-2, -1) / math.sqrt(k.size(-1))20 elif self.score == "general":21 s = q @ self.W(k).transpose(-2, -1)22 else: # additive: broadcast to (N, Lq, Lk, d_attn), the memory cost of this variant23 s = self.v(torch.tanh(self.Wq(q).unsqueeze(2) + self.Uk(k).unsqueeze(1))).squeeze(-1)24 if mask is not None:25 s = s.masked_fill(~mask, float("-inf")) # padded keys get weight exactly 026 a = s.softmax(dim=-1)27 return a @ v, a2829N, Lq, Lk, d = 2, 4, 7, 3230q, k, v = torch.randn(N, Lq, d), torch.randn(N, Lk, d), torch.randn(N, Lk, d)31lengths = torch.tensor([7, 5]) # second source has 2 padding positions32keep = (torch.arange(Lk)[None, :] < lengths[:, None])[:, None, :] # (N, 1, Lk)3334c, a = Attention(d, d, "scaled")(q, k, v, keep)35print(a.sum(-1)) # all ones: each query's weights form a distribution36print(a[1, :, 5:].abs().max()) # tensor(0.): padding never receives weight3738# Fused kernel (flash / memory-efficient backends when available); same maths39c2 = F.scaled_dot_product_attention(q, k, v, attn_mask=keep)40print(torch.allclose(c, c2, atol=1e-5)) # True4142# Learned projections + several heads; returns weights averaged over heads, shape (N, Lq, Lk)43mha = nn.MultiheadAttention(embed_dim=d, num_heads=4, batch_first=True)44out, w = mha(q, k, v, key_padding_mask=~keep[:, 0, :]) # here True = ignoreUse when
- +Removes the fixed-vector bottleneck: capacity grows with the input length
- +Short gradient path from every output to every input position
- +Fully differentiable and parallel over items; dot-product forms are one matrix multiply
- +Weights give an interpretable (if imperfect) alignment
Avoid when
- −Cost : quadratic when the sequence attends to itself
- −Order-blind: positions must be encoded in the keys and values
- −Saturated softmax (large unscaled scores) stops the scores from learning
- −Weights are easy to over-interpret as explanations
Check yourself
The query is almost identical to key A, but key B is three times longer and points roughly the same way. Which score function is most likely to put more weight on B?
Check yourself
With unscaled dot-product scores and d = 512, the softmax over 6 keys is almost one-hot. What happens to the gradient with respect to the scores?
Remember
- ✓Attention = score every key against the query, softmax, take the weighted average of the values: .
- ✓Score functions: dot , scaled dot , general , additive .
- ✓: saturated softmaxes stop learning, which is why dot products are divided by .
- ✓Soft attention is the expectation of hard attention; hard attention needs REINFORCE-style training.
- ✓Alignment matrices are useful diagnostics but not faithful explanations; mask padding so it gets zero weight.