# Attention mechanism

> A differentiable lookup: a query is scored against every key, a softmax turns the scores into weights, and the output is the weighted average of the values. The model reads whatever inputs matter at each step instead of squeezing them all into one fixed vector.

Source: https://www.mlfoundry.in/learn/attention-mechanism  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 13 min read · updated 2026-10-12

**Attention mechanism:** a layer that, given a **query** $q$ and a set of **key–value** pairs $(k_j, v_j)_{j=1}^{n}$, computes a score $s_j=\mathrm{score}(q,k_j)$ for every key, normalises the scores with a softmax into **attention weights** $a_j\ge0,\ \sum_j a_j=1$, and returns the **context vector** $c=\sum_j a_j v_j$. Every step is differentiable, so what to attend to is learned by backpropagation.

An **Encoder–decoder architecture** built from RNNs compresses the whole source into one fixed-size vector, and translation quality drops as sentences grow longer (Cho et al., 2014). Attention removes that bottleneck. The encoder keeps one vector per input position. At every output step the decoder forms a query from its current state and reads a fresh, input-dependent mixture of those vectors. Long inputs no longer have to fit into one vector, and gradients reach each input position through a short path instead of through the whole recurrence.

The ingredients appeared within a few years. Graves (2013) used a soft Gaussian window over the character string to let an RNN write handwriting for a given text. **Bahdanau, Cho and Bengio** (arXiv 2014, ICLR 2015) introduced the form used today for translation: an MLP scores each encoder annotation against the previous decoder state, and the softmax-weighted sum feeds the decoder. Graves, Wayne and Danihelka's Neural Turing Machine (2014) addressed an external memory by content (cosine similarity, sharpened by a key strength). **Luong, Pham and Manning** (2015) compared dot, general and concat scores and introduced local attention. Xu et al. (2015, "Show, Attend and Tell") separated **soft** from **hard** attention for image captioning. Vaswani et al. (2017) built the Transformer from **scaled dot-product attention** alone (see **Self-Attention**). Statistically, attention is a learned version of Nadaraya–Watson kernel regression (1964): predict at $q$ with a kernel-weighted average of stored outputs, $\hat y(q)=\sum_j K(q,k_j)\,y_j/\sum_{j'}K(q,k_{j'})$.

## Formula

$$
s_j=\mathrm{score}(q,k_j),\qquad a_j=\frac{\exp(s_j/\tau)}{\sum_{j'=1}^{n}\exp(s_{j'}/\tau)},\qquad c=\sum_{j=1}^{n}a_j\,v_j
$$

- $q$: query: what the current step is looking for (in RNN translation, the decoder state)
- $k_j$: key of item $j$: what the item is matched on (an encoder state, an image region, a memory slot)
- $v_j$: value of item $j$: what is read out if the item is attended to; often $v_j=k_j$ in RNN models, separate projections in Transformers
- $a_j$: attention (alignment) weight: a probability distribution over the $n$ items
- $\tau$: temperature, usually 1; $\tau\to0$ approaches a hard argmax, large $\tau$ a uniform average
- $c$: context vector: a convex combination of the values, so it lies inside their convex hull

With $n_q$ queries stacked as rows of $Q$, keys in $K$ and values in $V$, the whole layer is two matrix products around a row-wise softmax: $C=\mathrm{softmax}(S)\,V$ with $S_{ij}=\mathrm{score}(q_i,k_j)$. The matrix $A=\mathrm{softmax}(S)$ is the **alignment matrix**. Each row is one query's distribution over the items. Its cost is $O(n_q\,n)$ scores, which is why attention over long inputs is expensive.

## Gradient

The softmax Jacobian $\partial a_j/\partial s_l=a_j(\delta_{jl}-a_l)/\tau$ gives a compact rule for how the context moves when one score changes:

$$
\frac{\partial c}{\partial s_j}=\frac{a_j\,(v_j-c)}{\tau},\qquad \frac{\partial L}{\partial s_j}=\frac{a_j}{\tau}\Big(\frac{\partial L}{\partial c}\cdot v_j-\frac{\partial L}{\partial c}\cdot c\Big),\qquad \frac{\partial L}{\partial v_j}=a_j\,\frac{\partial L}{\partial c}
$$

- $a_j(v_j-c)$: raising $s_j$ pulls the context toward $v_j$, by an amount proportional to the weight $a_j$ already on it
- $\partial L/\partial v_j$: each value gets the upstream gradient scaled by its weight; items with $a_j\approx0$ learn nothing from this query

Two consequences follow. A score whose value equals the current context ($v_j=c$) has zero gradient. And once the softmax saturates (one $a_j\approx1$), every $a_j(v_j-c)$ is close to zero, so the scores stop learning. This is the reason for the $1/\sqrt d$ in scaled dot-product attention.

**At a glance**

- **Inputs:** one query (or $n_q$ of them), $n$ keys and $n$ values
- **Output:** a context vector per query, $c=\sum_j a_jv_j$, plus the weights $a$
- **Parameters:** none in the mechanism itself; the score function (general $W$, additive $W,U,v$) and the projections that produce $q,k,v$ are learned
- **Cost:** $O(n_q\,n\,d)$ time and $O(n_q\,n)$ memory for the weights
- **Order:** permutation-invariant over the items: positions only matter through what is encoded in $k_j$ and $v_j$
- **Kinds:** cross-attention (queries from the decoder, keys/values from the encoder), self-attention (all three from the same sequence), soft vs hard, global vs local

## The score function zoo

Everything except the score function is fixed. Four choices cover almost all RNN-era and Transformer models:

Classic score functions (q: query of size d_q, k: key of size d_k)

| name | score(q, k) | parameters | introduced in |
| --- | --- | --- | --- |
| dot | $q^\top k$ | none (needs $d_q=d_k$) | Luong et al. 2015 |
| scaled dot | $q^\top k/\sqrt{d_k}$ | none | Vaswani et al. 2017 |
| general (bilinear, multiplicative) | $q^\top W k$ | $W\in\mathbb R^{d_q\times d_k}$ | Luong et al. 2015 |
| additive (concat, MLP) | $v^\top\tanh(W q+U k)$ | $W,\ U,\ v$ with hidden size $d_a$ | Bahdanau et al. 2014 |

**Dot** says that a key matches when it points in the query's direction *and* is long. A long key wins even against a key that sits right next to the query. **General** inserts a learned matrix, so "match" can mean any linear relation between the two spaces, and it works across different sizes $d_q\ne d_k$. **Additive** feeds both vectors through a one-hidden-layer MLP with tanh, which can represent non-bilinear similarity. It was the original choice and is still common when $q$ and $k$ come from different networks. For large $d$, dot-product forms are much faster: the whole score matrix is one matrix multiply, while additive attention materialises an $n_q\times n\times d_a$ tensor of hidden units. Vaswani et al. note that the two perform similarly for small $d_k$, and that additive beats unscaled dot for large $d_k$. Their fix for dot is to scale it.

$$
q,k\sim\mathcal N(0,I_d)\ \Rightarrow\ \mathbb E[q^\top k]=0,\quad \operatorname{Var}(q^\top k)=\sum_{i=1}^{d}\operatorname{Var}(q_ik_i)=d\quad\Rightarrow\quad \operatorname{Var}\!\Big(\frac{q^\top k}{\sqrt d}\Big)=1
$$

- $\sqrt d$: the typical size of an unscaled score. Without the division, scores of size $\pm\sqrt d$ push the softmax toward a one-hot vector, where its gradients vanish

> Figure: Six keys in the plane, each with a scalar value v. Drag the query q: line width and circle area show the weights, and the table gives every score, weight and ∂c/∂s_j = a_j(v_j − c)/τ. At the start q = (0.90, 0.70) sits almost on top of 'cat', but the dot score prefers the longer key 'dog' (weights 0.434 vs 0.351, context c = 3.227). The hand-set additive scorer, whose vᵀtanh(Wq + Uk + b) peaks at q = k, puts 0.737 on 'cat' instead. 'general' with W a 90° rotation sends most weight to 'tea' (0.344). Lower τ to sharpen the weights (τ ≈ 0.25 gives 0.697 on 'dog'). 'sample 100' draws hard-attention picks from the same weights, and their mean value approaches the soft context. The bottom strip draws random q, k in d dimensions: the score's standard deviation grows like √d (23.09 at d = 512), and the mean largest of 6 weights rises from 0.446 at d = 2 to 0.954 at d = 512 unscaled, but stays between 0.37 and 0.43 after dividing by √d.

## Worked example: three keys, scaled dot product

Query $q=(1,\,0.5)$. Keys and values from the figure: cat $k_1=(1.0,\,0.8)$, $v_1=3$; dog $k_2=(1.7,\,0.2)$, $v_2=5$; car $k_3=(-1.0,\,1.1)$, $v_3=-2$. Here $d=2$, so scores are divided by $\sqrt2$.

Scaled dot-product attention by hand (checked numerically in Node)

| key | qᵀk | s = qᵀk/√2 | exp(s) | a | a·v | ∂c/∂s = a(v − c) |
| --- | --- | --- | --- | --- | --- | --- |
| cat | 1.40 | 0.9899 | 2.6911 | 0.3850 | 1.1551 | −0.1930 |
| dog | 1.80 | 1.2728 | 3.5708 | 0.5109 | 2.5543 | 0.7656 |
| car | −0.45 | −0.3182 | 0.7275 | 0.1041 | −0.2082 | −0.5726 |
| sum |  |  | 6.9894 | 1.0000 | $c=3.5014$ | 0.0000 |

**Reading the table**

1. Scores: $q^\top k_1=1\cdot1.0+0.5\cdot0.8=1.40$, $q^\top k_2=1.80$, $q^\top k_3=-0.45$; divide by $\sqrt2=1.4142$.
2. Softmax: $a_j=e^{s_j}/6.9894$, giving $(0.3850,\ 0.5109,\ 0.1041)$. Dog wins because its key is longest along $q$.
3. Context: $c=0.3850\cdot3+0.5109\cdot5+0.1041\cdot(-2)=3.5014$.
4. Gradient: raising dog's score moves $c$ toward 5 at rate $0.5109\,(5-3.5014)=0.7656$; raising car's moves it toward $-2$ at rate $-0.5726$. The three entries sum to 0 because adding the same constant to every score changes nothing.

## Soft versus hard attention

**Soft** (deterministic) attention returns the expectation $c=\mathbb E_{j\sim a}[v_j]$ and is trained by ordinary backprop. **Hard** (stochastic) attention samples one index $j\sim\mathrm{Categorical}(a)$ and returns $v_j$. Xu et al. (2015) compared both for captioning, with $a$ over the $14\times14=196$ locations of a VGG conv feature map. The sample is not differentiable, so they trained hard attention by maximising a variational lower bound on $\log p(y)$, which gives a REINFORCE-style estimator with a moving-average baseline and an entropy bonus. Mnih et al. (2014) had trained glimpse-based hard attention with REINFORCE the same way. Hard attention reads only one item per step, which is cheap at inference time, but its gradient estimate is noisy. Soft attention became the default; local attention (Luong et al., 2015) and sparse or top-$k$ variants sit in between.

> **The expectation link:** Hard attention's *expected* output is exactly the soft context: $\mathbb E_{j\sim a}[v_j]=\sum_j a_jv_j$. In the figure, 1000 hard draws at the default query average to 3.265 against the soft 3.227. Soft attention can be read as computing the expectation in closed form instead of sampling it. Bahdanau et al. describe their context as the "expected annotation" over possible alignments in exactly this sense.

## Alignments you can look at

For a cross-attention layer, the alignment matrix $A$ (rows: output steps, columns: input positions) is a soft word alignment. Bahdanau et al. plotted it for English→French and found mostly diagonal matrices with off-diagonal blocks where French reorders adjectives and nouns. In captioning (Xu et al.), each word's weights over the 196 locations, upsampled onto the image, highlight the region the word describes. Plots like these are a cheap sanity check: a model that attends uniformly, or always to the final token, has not learned to use attention.

> **Attention weights are not an explanation:** A large $a_j$ means $v_j$ contributed to $c$, not that item $j$ caused the prediction. Values can be small or cancel each other, later layers can ignore $c$, and different weight patterns can give the same output. Jain and Wallace (2019, "Attention is not Explanation") constructed adversarial weight distributions that leave predictions unchanged; Wiegreffe and Pinter (2019) argued the picture is more nuanced. Use attention maps to debug, and use gradient- or perturbation-based attribution when you need to know what the model actually relies on.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

class Attention(nn.Module):
    """query (N, Lq, dq), keys (N, Lk, dk), values (N, Lk, dv) -> context (N, Lq, dv), weights (N, Lq, Lk)"""
    def __init__(self, dq, dk, score="scaled", d_attn=64):
        super().__init__()
        self.score = score
        if score == "general":
            self.W = nn.Linear(dk, dq, bias=False)              # q^T W k
        if score == "additive":
            self.Wq = nn.Linear(dq, d_attn, bias=False)          # v^T tanh(W q + U k)
            self.Uk = nn.Linear(dk, d_attn, bias=False)
            self.v = nn.Linear(d_attn, 1, bias=False)

    def forward(self, q, k, v, mask=None):                       # mask: (N, Lq, Lk) or (N, 1, Lk), True = keep
        if self.score == "dot":
            s = q @ k.transpose(-2, -1)
        elif self.score == "scaled":
            s = q @ k.transpose(-2, -1) / math.sqrt(k.size(-1))
        elif self.score == "general":
            s = q @ self.W(k).transpose(-2, -1)
        else:  # additive: broadcast to (N, Lq, Lk, d_attn), the memory cost of this variant
            s = self.v(torch.tanh(self.Wq(q).unsqueeze(2) + self.Uk(k).unsqueeze(1))).squeeze(-1)
        if mask is not None:
            s = s.masked_fill(~mask, float("-inf"))              # padded keys get weight exactly 0
        a = s.softmax(dim=-1)
        return a @ v, a

N, Lq, Lk, d = 2, 4, 7, 32
q, k, v = torch.randn(N, Lq, d), torch.randn(N, Lk, d), torch.randn(N, Lk, d)
lengths = torch.tensor([7, 5])                                     # second source has 2 padding positions
keep = (torch.arange(Lk)[None, :] < lengths[:, None])[:, None, :]  # (N, 1, Lk)

c, a = Attention(d, d, "scaled")(q, k, v, keep)
print(a.sum(-1))                      # all ones: each query's weights form a distribution
print(a[1, :, 5:].abs().max())        # tensor(0.): padding never receives weight

# Fused kernel (flash / memory-efficient backends when available); same maths
c2 = F.scaled_dot_product_attention(q, k, v, attn_mask=keep)
print(torch.allclose(c, c2, atol=1e-5))  # True

# Learned projections + several heads; returns weights averaged over heads, shape (N, Lq, Lk)
mha = nn.MultiheadAttention(embed_dim=d, num_heads=4, batch_first=True)
out, w = mha(q, k, v, key_padding_mask=~keep[:, 0, :])    # here True = ignore
```

One attention layer with the four score functions, then the PyTorch built-ins

> **Masks: which way round?:** Padding positions must get weight 0, otherwise short sentences attend to garbage. Conventions differ: a boolean `attn_mask` in `F.scaled_dot_product_attention` means True = **take part**, while `key_padding_mask` in `nn.MultiheadAttention` means True = **ignore**. A row in which every key is masked becomes all $-\infty$, and the softmax returns NaN. Make sure every query keeps at least one key.

> **Where the variants are covered:** **Bahdanau attention** details the additive score inside an RNN encoder–decoder. **Luong attention** covers dot/general/concat, global versus local windows and input feeding. **Self-Attention** covers the case where queries, keys and values all come from the same sequence, with multi-head projections and positional information, as in the Transformer. At inference time an attentional decoder is usually run with **Beam search**.

**Strengths**

- Removes the fixed-vector bottleneck: capacity grows with the input length
- Short gradient path from every output to every input position
- Fully differentiable and parallel over items; dot-product forms are one matrix multiply
- Weights give an interpretable (if imperfect) alignment

**Limitations**

- Cost $O(n_q\,n)$: quadratic when the sequence attends to itself
- Order-blind: positions must be encoded in the keys and values
- Saturated softmax (large unscaled scores) stops the scores from learning
- Weights are easy to over-interpret as explanations

**Question:** The query is almost identical to key A, but key B is three times longer and points roughly the same way. Which score function is most likely to put more weight on B?

- A. Dot product
- B. Additive (MLP) with weights learned to peak at q = k
- C. Cosine similarity
- D. None: all score functions agree

**Answer:** A. Dot product — $q^\top k=\|q\|\|k\|\cos\theta$ grows with the key's length, so a long key in roughly the right direction can beat a short key that matches exactly. That is the 'dog' versus 'cat' case in the figure. Cosine similarity removes the length, and a learned additive scorer can represent closeness directly.

**Question:** With unscaled dot-product scores and d = 512, the softmax over 6 keys is almost one-hot. What happens to the gradient with respect to the scores?

- A. It becomes very large
- B. It becomes close to zero, because every a_j(v_j − c) is small
- C. It is unchanged
- D. It becomes NaN

**Answer:** B. It becomes close to zero, because every a_j(v_j − c) is small — $\partial c/\partial s_j=a_j(v_j-c)$. If one weight is near 1, then $c\approx v_{j^*}$ makes that term vanish, and every other $a_j\approx0$. Dividing scores by $\sqrt d$ brings their variance back to 1 (the figure shows the mean largest weight dropping from 0.954 to 0.429 at $d=512$).

**Key points**

- Attention = score every key against the query, softmax, take the weighted average of the values: $c=\sum_j a_jv_j$.
- Score functions: dot $q^\top k$, scaled dot $q^\top k/\sqrt d$, general $q^\top Wk$, additive $v^\top\tanh(Wq+Uk)$.
- $\partial c/\partial s_j=a_j(v_j-c)$: saturated softmaxes stop learning, which is why dot products are divided by $\sqrt d$.
- Soft attention is the expectation of hard attention; hard attention needs REINFORCE-style training.
- Alignment matrices are useful diagnostics but not faithful explanations; mask padding so it gets zero weight.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
