# Cross-attention

> Queries come from one sequence (decoder states, image latents) and keys and values from another (encoder output, text tokens): softmax(XW_Q(ZW_K)ᵀ/√d_k)ZW_V. The output has one row per query and costs n_q·m. Encoder–decoders, Stable Diffusion and Perceiver condition this way.

Source: https://www.mlfoundry.in/learn/cross-attention  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 13 min read · updated 2026-10-12

**Cross-attention:** (also **encoder–decoder attention**) is attention in which the queries come from one sequence $X\in\mathbb R^{n_q\times d_x}$ and the keys and values from a different sequence $Z\in\mathbb R^{m\times d_z}$, often called the **context** or **memory**: $Y=\mathrm{softmax}\big(XW_Q(ZW_K)^\top/\sqrt{d_k}\big)\,ZW_V$. Each of the $n_q$ positions of $X$ reads a weighted mixture of the $m$ positions of $Z$. The output has the shape of $X$, and $Z$ only supplies content.

It is the original use of attention. In the RNN translation models of **Bahdanau attention** (2014) and **Luong attention** (2015), the decoder state was the query and the encoder states were the keys and values: cross-attention in all but name. Vaswani et al. ("Attention Is All You Need", NeurIPS 2017) kept it as one of three attention sub-layers of the Transformer. In every decoder layer, "the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder", so every target position can attend over all source positions. The other two sub-layers are encoder **Self-attention** and masked decoder self-attention (see **Causal attention**).

Because the two sides need not share a modality, length or width, the same layer became the standard way to **condition** one model on another. T5 (Raffel et al., JMLR 2020) and Whisper are encoder–decoders that use it to read text or audio. Latent diffusion (Rombach et al., CVPR 2022, the basis of Stable Diffusion) injects a text prompt into a U-Net by letting image positions query text-encoder states. DETR (Carion et al., ECCV 2020) decodes 100 learned object queries against CNN image features. Perceiver (Jaegle et al., ICML 2021) uses a small latent array to query a very large input. Flamingo (Alayrac et al., NeurIPS 2022) lets a frozen language model attend to visual tokens through new gated cross-attention layers.

## The layer

$$
Q=XW_Q,\quad K=ZW_K,\quad V=ZW_V,\qquad A=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}+M\Big)\in\mathbb R^{n_q\times m},\qquad Y=AV\in\mathbb R^{n_q\times d_v}
$$

- $X\in\mathbb R^{n_q\times d_x}$: the stream being updated: decoder states, image latents, object queries, a latent array
- $Z\in\mathbb R^{m\times d_z}$: the context: encoder output, text-encoder states, image features, raw input bytes
- $W_Q\in\mathbb R^{d_x\times d_k},\ W_K\in\mathbb R^{d_z\times d_k}$: map both sides into a shared $d_k$-dimensional matching space; $d_x\ne d_z$ is allowed
- $A$: a rectangular $n_q\times m$ matrix; row $i$ is the distribution of query $i$ over the $m$ context positions
- $M$: usually only a **padding** mask on the context; no causal mask, since the context is fully known

The arithmetic is exactly **Scaled dot-product attention**. What changes is the roles. The output length follows the queries, so the layer can turn $m$ context tokens into any number $n_q$ of outputs: 4096 latent pixels from 77 text tokens, 100 object slots from thousands of image features, or 512 latents from 50,176 pixels. The cost is $O(n_q\,m\,d_k)$ time and $n_q\times m$ weights per head, linear in each length. Self-attention over the concatenation $[X;Z]$ would cost $(n_q+m)^2$.

In a Transformer decoder layer the order is self-attention, then cross-attention, then the feed-forward block, each with a residual connection and layer norm. The residual matters: cross-attention writes context information **into** the query stream, $x\leftarrow x+\mathrm{CrossAttn}(x,Z)$, and the model can learn to ignore it where the context is irrelevant. Flamingo makes this explicit with a gate: $x\leftarrow x+\tanh(\alpha)\,\mathrm{CrossAttn}(x,Z)$ with $\alpha$ initialised at 0, so the frozen language model starts out exactly unchanged.

## Where the queries and the keys come from

Cross-attention in widely used models

| model | queries (X) | keys and values (Z) | notes |
| --- | --- | --- | --- |
| Transformer, T5 (encoder–decoder) | decoder states | final encoder output | every decoder layer attends to the same encoder output |
| Whisper | text-token decoder states | audio encoder output, 1500 positions for 30 s | speech recognition and translation |
| Latent diffusion / Stable Diffusion v1 | U-Net latent positions, $64\times64=4096$ at the top level for $512\times512$ images | frozen CLIP ViT-L/14 text encoder, 77 tokens × 768 | $Q=W_Q\varphi(z_t)$, $K,V$ from $\tau_\theta(y)$ (Rombach et al. 2022) |
| DETR | 100 learned object queries | CNN feature map plus positional encodings | each query decodes one box or 'no object' |
| Perceiver | latent array, $N=512$ for ImageNet | input array, $M=224^2=50{,}176$ pixels | cost $O(MN)$, then self-attention among latents only |
| Flamingo | frozen language-model tokens | visual tokens from a Perceiver Resampler (64 per image) | gated: $\tanh(\alpha)$, $\alpha=0$ at initialisation |

Perceiver turns the usual picture around. Instead of a short query sequence reading a long context because the task requires it, it chooses a short learned latent array $N\ll M$ so that the expensive part, attention over the input, is linear in $M$. Only the small latent array then goes through deep self-attention at $O(N^2)$ per layer. Perceiver IO (ICLR 2022) adds a second cross-attention at the output, where task-specific output queries read the latents. Many efficient designs, such as **Multimodal transformers** with resampler modules and **Retrieval-augmented transformers** that attend to retrieved chunks, reuse this pattern of a small set of queries reading a large memory.

> Figure: Text-to-image cross-attention in the style of a diffusion U-Net. A 10 × 10 grid of image latents provides 100 queries. The prompt '<sot> a red ball on green grass' provides 7 keys and values. Latent features and keys are hand-set (object, ground, background), and each value writes a colour. The selected latent in the ball puts 0.550 on 'red' and 0.391 on 'ball'. A ground latent puts 0.550 on 'grass' and 0.364 on 'green'. A sky latent gives 0.826 to the start token <sot>, which acts as an attention sink and takes 0.483 of the mean weight over the grid. Switch 'red' to 'blue' or 'yellow': the ball changes colour, but the colour word's key barely moves, so no attention weight changes by more than 0.039 (blue) or 0.004 (yellow). Keys decide where a word acts and values decide what it writes, which is the observation behind Prompt-to-Prompt editing. Resize the grid from 6 × 6 to 16 × 16: the output always has one vector per latent, and the score count n_q·m grows from 252 to 1,792, against 69,169 for self-attention over all 263 latents and tokens. At Stable Diffusion v1's top resolution, 4096 latents × 77 tokens give 315,392 cross-attention scores per head, against 16,777,216 for the latents' own self-attention.

## Worked example: two queries, three context tokens

Decoder queries (already projected) $q_1=(1,0)$, $q_2=(0,1)$. Context keys $k_1=(1,1)$, $k_2=(2,0)$, $k_3=(0,2)$ and values $v_1=(1,0)$, $v_2=(0,1)$, $v_3=(1,1)$, with $d_k=2$. In a second batch the third context position is padding.

A 2 × 3 attention matrix, without and with a padding mask on context position 3 (checked in PyTorch)

| query | scores qᵀkⱼ/√2 | A | output | A (k₃ padded) | output (k₃ padded) |
| --- | --- | --- | --- | --- | --- |
| q₁ = (1, 0) | 0.7071, 1.4142, 0 | 0.2840, 0.5760, 0.1400 | (0.4240, 0.7160) | 0.3302, 0.6698, 0 | (0.3302, 0.6698) |
| q₂ = (0, 1) | 0.7071, 0, 1.4142 | 0.2840, 0.1400, 0.5760 | (0.8600, 0.7160) | 0.6698, 0.3302, 0 | (0.6698, 0.3302) |

**Reading the table**

1. $A$ is $2\times3$: two queries, three context positions. The output is $2\times2$, one row per query, whatever $m$ is.
2. $q_1$ matches $k_2$ best ($2/\sqrt2=1.4142$) and $q_2$ matches $k_3$; both match $k_1$ equally. Output 1: $0.284\,(1,0)+0.576\,(0,1)+0.14\,(1,1)=(0.4240,\,0.7160)$.
3. Padding $k_3$ sets its score to $-\infty$. Row 2 renormalises over $k_1,k_2$: $e^{0.7071}/(e^{0.7071}+e^{0})=0.6698$.
4. The keys and values depend only on $Z$. When the decoder produces one more token, only a new query row is added; $K$ and $V$ are computed once per source sentence and reused at every step.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class CrossAttention(nn.Module):
    """Queries from x (n_q tokens, width d_model); keys and values from context (m tokens, width d_ctx)."""
    def __init__(self, d_model, d_ctx, heads=8, d_head=64):
        super().__init__()
        self.h, inner = heads, heads * d_head
        self.to_q = nn.Linear(d_model, inner, bias=False)
        self.to_k = nn.Linear(d_ctx, inner, bias=False)      # context may have a different width
        self.to_v = nn.Linear(d_ctx, inner, bias=False)
        self.to_out = nn.Linear(inner, d_model)

    def kv(self, context):
        # computed once per context; reused at every decoding step
        split = lambda t: t.unflatten(-1, (self.h, -1)).transpose(1, 2)   # (N, h, m, d_head)
        return split(self.to_k(context)), split(self.to_v(context))

    def forward(self, x, context=None, kv=None, context_mask=None):
        # x (N, n_q, d_model); context (N, m, d_ctx); context_mask (N, m), True = real token
        k, v = kv if kv is not None else self.kv(context)
        q = self.to_q(x).unflatten(-1, (self.h, -1)).transpose(1, 2)        # (N, h, n_q, d_head)
        mask = None if context_mask is None else context_mask[:, None, None, :]   # (N, 1, 1, m)
        y = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)        # (N, h, n_q, d_head)
        return self.to_out(y.transpose(1, 2).flatten(2))                   # (N, n_q, d_model)

torch.manual_seed(0)
# Text-to-image, Stable Diffusion v1 sizes: 64x64 latent positions at width 320, 77 CLIP tokens of width 768
N, n_q, m = 1, 64 * 64, 77
x = torch.randn(N, n_q, 320)
text = torch.randn(N, m, 768)
attn = CrossAttention(d_model=320, d_ctx=768, heads=8, d_head=40)
print(attn(x, text).shape)                       # torch.Size([1, 4096, 320]): one output per query

# Encoder-decoder: a padded source, decoder queries attend to the real source tokens only
src = torch.randn(2, 12, 512)
src_mask = torch.arange(12)[None, :] < torch.tensor([[12], [7]])     # (2, 12)
xattn = CrossAttention(512, 512)
kv = xattn.kv(src)                                # K, V cached once: 2 x (2, 8, 12, 64)
tgt = torch.randn(2, 5, 512)
full = xattn(tgt, kv=kv, context_mask=src_mask)   # all 5 target positions (teacher forcing)
step = xattn(tgt[:, -1:], kv=kv, context_mask=src_mask)            # one new decoding step
print(full.shape, step.shape)                     # torch.Size([2, 5, 512]) torch.Size([2, 1, 512])
print(torch.allclose(full[:, -1:], step, atol=1e-5))                 # True: no causal mask needed

# Changing a padded source position has no effect on sequence 1
src2 = src.clone(); src2[1, 9] += 100.0
print(torch.allclose(xattn(tgt, src2, context_mask=src_mask), full, atol=1e-5))   # True
```

A multi-head cross-attention module with a context of different width, a padding mask and cached keys and values

> **Do not put a causal mask on the context:** The context is fully available before decoding starts, so every decoder position may attend to every real source position. The only mask cross-attention needs is the source **padding** mask, of shape $(N,1,1,m)$ broadcast over heads and queries. Reusing the decoder's causal mask here is a common bug. With $n_q\ne m$ it often fails on shapes; with $n_q=m$ it silently hides the second half of the source from the first target tokens.

> **Which side is the query?:** The output has the length of the **queries**. If you want to update image features with text, the image is $X$ and the text is $Z$; swapping them gives one output per text token instead. In `nn.MultiheadAttention(embed_dim, num_heads, kdim=..., vdim=...)` the call is `attn(query, key, value)` with `key` and `value` both set to the context, and `kdim`/`vdim` are needed when the context width differs from `embed_dim`.

> **Keys choose where, values choose what:** In a cross-attention map, the weights from image positions to one prompt token show where that token acts. Hertz et al. ("Prompt-to-Prompt", ICLR 2023) observed that these maps largely fix the layout of a generated image, and edited images by keeping the maps and changing the words: the values change, the layout stays. The same separation makes encoder–decoder inference cheap. $K$ and $V$ of the source are computed once, and each new target token only adds one query.

> **The start token as an attention sink:** Softmax weights must sum to 1, so a query that finds nothing relevant must still put its weight somewhere. In Stable Diffusion, much of the cross-attention mass goes to the start-of-text token, whose CLIP state carries little word-specific content. The figure reproduces this: background latents give 0.826 to <sot>. When you visualise per-word maps, normalise them per token or exclude the start token, or the sink dominates every plot.

**Strengths**

- Conditions any stream on any other: different lengths, widths and modalities
- Cost $O(n_q\,m)$ rather than $O((n_q+m)^2)$; with few queries it is linear in the context length (Perceiver)
- Context keys and values are computed once and cached for every decoding step
- Attention maps give an interpretable soft alignment between the two inputs

**Limitations**

- The context side is not updated by the layer; information flows one way per layer
- Every decoder layer stores its own $K,V$ of the context: memory $2\,L\,m\,d$ per sequence
- With few queries (Perceiver latents) the bottleneck can drop input detail
- Mask conventions and query/key roles are easy to get backwards

**Question:** A Stable Diffusion U-Net block cross-attends from 4096 latent positions to 77 text tokens. What are the shapes of the attention matrix per head and of the output?

- A. 77 × 4096 and 77 × d
- B. 4096 × 77 and 4096 × d
- C. 4096 × 4096 and 4096 × d
- D. (4096 + 77) × (4096 + 77) and 4173 × d

**Answer:** B. 4096 × 77 and 4096 × d — Queries come from the image latents, so $A\in\mathbb R^{n_q\times m}=4096\times77$, and the output has one row per query: $4096\times d$, the same shape as the latents it is added to. The text length only sets the number of columns.

**Question:** During greedy decoding with an encoder–decoder Transformer, which parts of a cross-attention layer must be recomputed at each new target step?

- A. Keys and values of the source, and the query of the new token
- B. Only the query of the new token (and its attention over the cached source keys and values)
- C. The whole attention matrix for all previous target tokens
- D. Nothing; cross-attention is computed once per sentence

**Answer:** B. Only the query of the new token (and its attention over the cached source keys and values) — $K=ZW_K$ and $V=ZW_V$ depend only on the encoder output $Z$, which does not change during decoding. Each step adds one query row, $q_tK^\top$ over the $m$ cached keys. The decoder's self-attention, by contrast, grows its own key–value cache by one entry per step.

**Key points**

- $Y=\mathrm{softmax}\big(XW_Q(ZW_K)^\top/\sqrt{d_k}\big)\,ZW_V$: queries from $X$ ($n_q$ rows), keys and values from the context $Z$ ($m$ rows); $A\in\mathbb R^{n_q\times m}$.
- Output has one row per query; cost $O(n_q\,m\,d_k)$, linear in each length; only a context padding mask, never a causal one.
- Encoder–decoder Transformers and T5 cross-attend from every decoder layer to the final encoder output; K and V are cached once per source.
- Stable Diffusion: U-Net latents query 77 CLIP text tokens; Perceiver: a small latent array queries a huge input; DETR: 100 object queries read image features.
- Keys decide where a context token is read, values decide what is written, and a residual (or Flamingo's tanh gate) adds it to the query stream.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
