Attention mechanism

A differentiable lookup: a query is scored against every key, a softmax turns the scores into weights, and the output is the weighted average of the values. The model reads whatever inputs matter at each step instead of squeezing them all into one fixed vector.

13 min read1222 wordscore2 viewsupdated 2026-10-12

Definition

Attention mechanism a layer that, given a query qq and a set of key–value pairs (kj,vj)j=1n(k_j, v_j)_{j=1}^{n}, computes a score sj=score(q,kj)s_j=\mathrm{score}(q,k_j) for every key, normalises the scores with a softmax into attention weights aj≥0, ∑jaj=1a_j\ge0,\ \sum_j a_j=1, and returns the context vector c=∑jajvjc=\sum_j a_j v_j. Every step is differentiable, so what to attend to is learned by backpropagation.

An Encoder–decoder architecture built from RNNs compresses the whole source into one fixed-size vector, and translation quality drops as sentences grow longer (Cho et al., 2014). Attention removes that bottleneck. The encoder keeps one vector per input position. At every output step the decoder forms a query from its current state and reads a fresh, input-dependent mixture of those vectors. Long inputs no longer have to fit into one vector, and gradients reach each input position through a short path instead of through the whole recurrence.

The ingredients appeared within a few years. Graves (2013) used a soft Gaussian window over the character string to let an RNN write handwriting for a given text. Bahdanau, Cho and Bengio (arXiv 2014, ICLR 2015) introduced the form used today for translation: an MLP scores each encoder annotation against the previous decoder state, and the softmax-weighted sum feeds the decoder. Graves, Wayne and Danihelka's Neural Turing Machine (2014) addressed an external memory by content (cosine similarity, sharpened by a key strength). Luong, Pham and Manning (2015) compared dot, general and concat scores and introduced local attention. Xu et al. (2015, "Show, Attend and Tell") separated soft from hard attention for image captioning. Vaswani et al. (2017) built the Transformer from scaled dot-product attention alone (see Self-Attention). Statistically, attention is a learned version of Nadaraya–Watson kernel regression (1964): predict at qq with a kernel-weighted average of stored outputs, y^(q)=∑jK(q,kj) yj/∑j′K(q,kj′)\hat y(q)=\sum_j K(q,k_j)\,y_j/\sum_{j'}K(q,k_{j'}).

Formula

sj=score(q,kj),aj=exp⁡(sj/τ)∑j′=1nexp⁡(sj′/τ),c=∑j=1naj vjs_j=\mathrm{score}(q,k_j),\qquad a_j=\frac{\exp(s_j/\tau)}{\sum_{j'=1}^{n}\exp(s_{j'}/\tau)},\qquad c=\sum_{j=1}^{n}a_j\,v_j
qq
query: what the current step is looking for (in RNN translation, the decoder state)
kjk_j
key of item jj: what the item is matched on (an encoder state, an image region, a memory slot)
vjv_j
value of item jj: what is read out if the item is attended to; often vj=kjv_j=k_j in RNN models, separate projections in Transformers
aja_j
attention (alignment) weight: a probability distribution over the nn items
τ\tau
temperature, usually 1; τ→0\tau\to0 approaches a hard argmax, large τ\tau a uniform average
cc
context vector: a convex combination of the values, so it lies inside their convex hull

With nqn_q queries stacked as rows of QQ, keys in KK and values in VV, the whole layer is two matrix products around a row-wise softmax: C=softmax(S) VC=\mathrm{softmax}(S)\,V with Sij=score(qi,kj)S_{ij}=\mathrm{score}(q_i,k_j). The matrix A=softmax(S)A=\mathrm{softmax}(S) is the alignment matrix. Each row is one query's distribution over the items. Its cost is O(nq n)O(n_q\,n) scores, which is why attention over long inputs is expensive.

Gradient

The softmax Jacobian ∂aj/∂sl=aj(δjl−al)/τ\partial a_j/\partial s_l=a_j(\delta_{jl}-a_l)/\tau gives a compact rule for how the context moves when one score changes:

∂c∂sj=aj (vj−c)τ,∂L∂sj=ajτ(∂L∂c⋅vj−∂L∂c⋅c),∂L∂vj=aj ∂L∂c\frac{\partial c}{\partial s_j}=\frac{a_j\,(v_j-c)}{\tau},\qquad \frac{\partial L}{\partial s_j}=\frac{a_j}{\tau}\Big(\frac{\partial L}{\partial c}\cdot v_j-\frac{\partial L}{\partial c}\cdot c\Big),\qquad \frac{\partial L}{\partial v_j}=a_j\,\frac{\partial L}{\partial c}
aj(vj−c)a_j(v_j-c)
raising sjs_j pulls the context toward vjv_j, by an amount proportional to the weight aja_j already on it
∂L/∂vj\partial L/\partial v_j
each value gets the upstream gradient scaled by its weight; items with aj≈0a_j\approx0 learn nothing from this query

Two consequences follow. A score whose value equals the current context (vj=cv_j=c) has zero gradient. And once the softmax saturates (one aj≈1a_j\approx1), every aj(vj−c)a_j(v_j-c) is close to zero, so the scores stop learning. This is the reason for the 1/d1/\sqrt d in scaled dot-product attention.

At a glance

Inputs
one query (or nqn_q of them), nn keys and nn values
Output
a context vector per query, c=∑jajvjc=\sum_j a_jv_j, plus the weights aa
Parameters
none in the mechanism itself; the score function (general WW, additive W,U,vW,U,v) and the projections that produce q,k,vq,k,v are learned
Cost
O(nq n d)O(n_q\,n\,d) time and O(nq n)O(n_q\,n) memory for the weights
Order
permutation-invariant over the items: positions only matter through what is encoded in kjk_j and vjv_j
Kinds
cross-attention (queries from the decoder, keys/values from the encoder), self-attention (all three from the same sequence), soft vs hard, global vs local

The score function zoo

Everything except the score function is fixed. Four choices cover almost all RNN-era and Transformer models:

namescore(q, k)parametersintroduced in
dotq⊤kq^\top knone (needs dq=dkd_q=d_k)Luong et al. 2015
scaled dotq⊤k/dkq^\top k/\sqrt{d_k}noneVaswani et al. 2017
general (bilinear, multiplicative)q⊤Wkq^\top W kW∈Rdq×dkW\in\mathbb R^{d_q\times d_k}Luong et al. 2015
additive (concat, MLP)v⊤tanh⁡(Wq+Uk)v^\top\tanh(W q+U k)W, U, vW,\ U,\ v with hidden size dad_aBahdanau et al. 2014
Classic score functions (q: query of size d_q, k: key of size d_k)

Dot says that a key matches when it points in the query's direction and is long. A long key wins even against a key that sits right next to the query. General inserts a learned matrix, so "match" can mean any linear relation between the two spaces, and it works across different sizes dq≠dkd_q\ne d_k. Additive feeds both vectors through a one-hidden-layer MLP with tanh, which can represent non-bilinear similarity. It was the original choice and is still common when qq and kk come from different networks. For large dd, dot-product forms are much faster: the whole score matrix is one matrix multiply, while additive attention materialises an nq×n×dan_q\times n\times d_a tensor of hidden units. Vaswani et al. note that the two perform similarly for small dkd_k, and that additive beats unscaled dot for large dkd_k. Their fix for dot is to scale it.

q,k∼N(0,Id) ⇒ E[q⊤k]=0,Var⁡(q⊤k)=∑i=1dVar⁡(qiki)=d⇒Var⁡ ⁣(q⊤kd)=1q,k\sim\mathcal N(0,I_d)\ \Rightarrow\ \mathbb E[q^\top k]=0,\quad \operatorname{Var}(q^\top k)=\sum_{i=1}^{d}\operatorname{Var}(q_ik_i)=d\quad\Rightarrow\quad \operatorname{Var}\!\Big(\frac{q^\top k}{\sqrt d}\Big)=1
d\sqrt d
the typical size of an unscaled score. Without the division, scores of size ±d\pm\sqrt d push the softmax toward a one-hot vector, where its gradients vanish
drag the query q · sⱼ = qᵀkⱼcatv=3dogv=5carv=-2busv=-4teav=1jamv=2qq = (0.90, 0.70) · circle area ∝ weight · line width ∝ weightkeyscore sⱼweight aⱼ∂c/∂sⱼcat 1.460.351dog 1.670.434car−0.130.072bus−1.560.017tea−0.820.036jam 0.100.090soft: c = Σ aⱼvⱼ = 3.227 · τ = 1.00entropy 1.91 bits (uniform 2.58)hard: no samples yetE[v_j], j ~ a, equals the soft contextwhy √d: q, kⱼ ~ N(0, I_d), d = 64, 6 keys, 300 draws · std of qᵀk = 7.99 (√d = 8.00)mean largest weight: unscaled 0.899 · divided by √d 0.421 (uniform = 0.167)unscaled scores grow like √d, so the softmax saturates toward one-hot and its gradients vanish
score
hard
fig.Six keys in the plane, each with a scalar value v. Drag the query q: line width and circle area show the weights, and the table gives every score, weight and ∂c/∂s_j = a_j(v_j − c)/τ. At the start q = (0.90, 0.70) sits almost on top of 'cat', but the dot score prefers the longer key 'dog' (weights 0.434 vs 0.351, context c = 3.227). The hand-set additive scorer, whose vᵀtanh(Wq + Uk + b) peaks at q = k, puts 0.737 on 'cat' instead. 'general' with W a 90° rotation sends most weight to 'tea' (0.344). Lower τ to sharpen the weights (τ ≈ 0.25 gives 0.697 on 'dog'). 'sample 100' draws hard-attention picks from the same weights, and their mean value approaches the soft context. The bottom strip draws random q, k in d dimensions: the score's standard deviation grows like √d (23.09 at d = 512), and the mean largest of 6 weights rises from 0.446 at d = 2 to 0.954 at d = 512 unscaled, but stays between 0.37 and 0.43 after dividing by √d.

Worked example: three keys, scaled dot product

Query q=(1, 0.5)q=(1,\,0.5). Keys and values from the figure: cat k1=(1.0, 0.8)k_1=(1.0,\,0.8), v1=3v_1=3; dog k2=(1.7, 0.2)k_2=(1.7,\,0.2), v2=5v_2=5; car k3=(−1.0, 1.1)k_3=(-1.0,\,1.1), v3=−2v_3=-2. Here d=2d=2, so scores are divided by 2\sqrt2.

keyqᵀks = qᵀk/√2exp(s)aa·v∂c/∂s = a(v − c)
cat1.400.98992.69110.38501.1551−0.1930
dog1.801.27283.57080.51092.55430.7656
car−0.45−0.31820.72750.1041−0.2082−0.5726
sum6.98941.0000c=3.5014c=3.50140.0000
Scaled dot-product attention by hand (checked numerically in Node)

Reading the table

  1. 1Scores: q⊤k1=1⋅1.0+0.5⋅0.8=1.40q^\top k_1=1\cdot1.0+0.5\cdot0.8=1.40, q⊤k2=1.80q^\top k_2=1.80, q⊤k3=−0.45q^\top k_3=-0.45; divide by 2=1.4142\sqrt2=1.4142.
  2. 2Softmax: aj=esj/6.9894a_j=e^{s_j}/6.9894, giving (0.3850, 0.5109, 0.1041)(0.3850,\ 0.5109,\ 0.1041). Dog wins because its key is longest along qq.
  3. 3Context: c=0.3850⋅3+0.5109⋅5+0.1041⋅(−2)=3.5014c=0.3850\cdot3+0.5109\cdot5+0.1041\cdot(-2)=3.5014.
  4. 4Gradient: raising dog's score moves cc toward 5 at rate 0.5109 (5−3.5014)=0.76560.5109\,(5-3.5014)=0.7656; raising car's moves it toward −2-2 at rate −0.5726-0.5726. The three entries sum to 0 because adding the same constant to every score changes nothing.

Soft versus hard attention

Soft (deterministic) attention returns the expectation c=Ej∼a[vj]c=\mathbb E_{j\sim a}[v_j] and is trained by ordinary backprop. Hard (stochastic) attention samples one index j∼Categorical(a)j\sim\mathrm{Categorical}(a) and returns vjv_j. Xu et al. (2015) compared both for captioning, with aa over the 14×14=19614\times14=196 locations of a VGG conv feature map. The sample is not differentiable, so they trained hard attention by maximising a variational lower bound on log⁡p(y)\log p(y), which gives a REINFORCE-style estimator with a moving-average baseline and an entropy bonus. Mnih et al. (2014) had trained glimpse-based hard attention with REINFORCE the same way. Hard attention reads only one item per step, which is cheap at inference time, but its gradient estimate is noisy. Soft attention became the default; local attention (Luong et al., 2015) and sparse or top-kk variants sit in between.

Alignments you can look at

For a cross-attention layer, the alignment matrix AA (rows: output steps, columns: input positions) is a soft word alignment. Bahdanau et al. plotted it for English→French and found mostly diagonal matrices with off-diagonal blocks where French reorders adjectives and nouns. In captioning (Xu et al.), each word's weights over the 196 locations, upsampled onto the image, highlight the region the word describes. Plots like these are a cheap sanity check: a model that attends uniformly, or always to the final token, has not learned to use attention.

python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3class Attention(nn.Module):
4 """query (N, Lq, dq), keys (N, Lk, dk), values (N, Lk, dv) -> context (N, Lq, dv), weights (N, Lq, Lk)"""
5 def __init__(self, dq, dk, score="scaled", d_attn=64):
6 super().__init__()
7 self.score = score
8 if score == "general":
9 self.W = nn.Linear(dk, dq, bias=False) # q^T W k
10 if score == "additive":
11 self.Wq = nn.Linear(dq, d_attn, bias=False) # v^T tanh(W q + U k)
12 self.Uk = nn.Linear(dk, d_attn, bias=False)
13 self.v = nn.Linear(d_attn, 1, bias=False)
14
15 def forward(self, q, k, v, mask=None): # mask: (N, Lq, Lk) or (N, 1, Lk), True = keep
16 if self.score == "dot":
17 s = q @ k.transpose(-2, -1)
18 elif self.score == "scaled":
19 s = q @ k.transpose(-2, -1) / math.sqrt(k.size(-1))
20 elif self.score == "general":
21 s = q @ self.W(k).transpose(-2, -1)
22 else: # additive: broadcast to (N, Lq, Lk, d_attn), the memory cost of this variant
23 s = self.v(torch.tanh(self.Wq(q).unsqueeze(2) + self.Uk(k).unsqueeze(1))).squeeze(-1)
24 if mask is not None:
25 s = s.masked_fill(~mask, float("-inf")) # padded keys get weight exactly 0
26 a = s.softmax(dim=-1)
27 return a @ v, a
28
29N, Lq, Lk, d = 2, 4, 7, 32
30q, k, v = torch.randn(N, Lq, d), torch.randn(N, Lk, d), torch.randn(N, Lk, d)
31lengths = torch.tensor([7, 5]) # second source has 2 padding positions
32keep = (torch.arange(Lk)[None, :] < lengths[:, None])[:, None, :] # (N, 1, Lk)
33
34c, a = Attention(d, d, "scaled")(q, k, v, keep)
35print(a.sum(-1)) # all ones: each query's weights form a distribution
36print(a[1, :, 5:].abs().max()) # tensor(0.): padding never receives weight
37
38# Fused kernel (flash / memory-efficient backends when available); same maths
39c2 = F.scaled_dot_product_attention(q, k, v, attn_mask=keep)
40print(torch.allclose(c, c2, atol=1e-5)) # True
41
42# Learned projections + several heads; returns weights averaged over heads, shape (N, Lq, Lk)
43mha = nn.MultiheadAttention(embed_dim=d, num_heads=4, batch_first=True)
44out, w = mha(q, k, v, key_padding_mask=~keep[:, 0, :]) # here True = ignore
One attention layer with the four score functions, then the PyTorch built-ins

Use when

  • +Removes the fixed-vector bottleneck: capacity grows with the input length
  • +Short gradient path from every output to every input position
  • +Fully differentiable and parallel over items; dot-product forms are one matrix multiply
  • +Weights give an interpretable (if imperfect) alignment

Avoid when

  • −Cost O(nq n)O(n_q\,n): quadratic when the sequence attends to itself
  • −Order-blind: positions must be encoded in the keys and values
  • −Saturated softmax (large unscaled scores) stops the scores from learning
  • −Weights are easy to over-interpret as explanations

Check yourself

The query is almost identical to key A, but key B is three times longer and points roughly the same way. Which score function is most likely to put more weight on B?

Check yourself

With unscaled dot-product scores and d = 512, the softmax over 6 keys is almost one-hot. What happens to the gradient with respect to the scores?

Remember

  • ✓Attention = score every key against the query, softmax, take the weighted average of the values: c=∑jajvjc=\sum_j a_jv_j.
  • ✓Score functions: dot q⊤kq^\top k, scaled dot q⊤k/dq^\top k/\sqrt d, general q⊤Wkq^\top Wk, additive v⊤tanh⁡(Wq+Uk)v^\top\tanh(Wq+Uk).
  • ✓∂c/∂sj=aj(vj−c)\partial c/\partial s_j=a_j(v_j-c): saturated softmaxes stop learning, which is why dot products are divided by d\sqrt d.
  • ✓Soft attention is the expectation of hard attention; hard attention needs REINFORCE-style training.
  • ✓Alignment matrices are useful diagnostics but not faithful explanations; mask padding so it gets zero weight.
← back to Deep Learning roadmap