Cross-attention

Queries come from one sequence (decoder states, image latents) and keys and values from another (encoder output, text tokens): softmax(XW_Q(ZW_K)ᵀ/√d_k)ZW_V. The output has one row per query and costs n_q·m. Encoder–decoders, Stable Diffusion and Perceiver condition this way.

13 min read1057 wordscore0 viewsupdated 2026-10-12

Definition

Cross-attention (also encoder–decoder attention) is attention in which the queries come from one sequence X∈Rnq×dxX\in\mathbb R^{n_q\times d_x} and the keys and values from a different sequence Z∈Rm×dzZ\in\mathbb R^{m\times d_z}, often called the context or memory: Y=softmax(XWQ(ZWK)⊤/dk) ZWVY=\mathrm{softmax}\big(XW_Q(ZW_K)^\top/\sqrt{d_k}\big)\,ZW_V. Each of the nqn_q positions of XX reads a weighted mixture of the mm positions of ZZ. The output has the shape of XX, and ZZ only supplies content.

It is the original use of attention. In the RNN translation models of Bahdanau attention (2014) and Luong attention (2015), the decoder state was the query and the encoder states were the keys and values: cross-attention in all but name. Vaswani et al. ("Attention Is All You Need", NeurIPS 2017) kept it as one of three attention sub-layers of the Transformer. In every decoder layer, "the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder", so every target position can attend over all source positions. The other two sub-layers are encoder Self-attention and masked decoder self-attention (see Causal attention).

Because the two sides need not share a modality, length or width, the same layer became the standard way to condition one model on another. T5 (Raffel et al., JMLR 2020) and Whisper are encoder–decoders that use it to read text or audio. Latent diffusion (Rombach et al., CVPR 2022, the basis of Stable Diffusion) injects a text prompt into a U-Net by letting image positions query text-encoder states. DETR (Carion et al., ECCV 2020) decodes 100 learned object queries against CNN image features. Perceiver (Jaegle et al., ICML 2021) uses a small latent array to query a very large input. Flamingo (Alayrac et al., NeurIPS 2022) lets a frozen language model attend to visual tokens through new gated cross-attention layers.

The layer

Q=XWQ,K=ZWK,V=ZWV,A=softmax ⁣(QK⊤dk+M)∈Rnq×m,Y=AV∈Rnq×dvQ=XW_Q,\quad K=ZW_K,\quad V=ZW_V,\qquad A=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}+M\Big)\in\mathbb R^{n_q\times m},\qquad Y=AV\in\mathbb R^{n_q\times d_v}
X∈Rnq×dxX\in\mathbb R^{n_q\times d_x}
the stream being updated: decoder states, image latents, object queries, a latent array
Z∈Rm×dzZ\in\mathbb R^{m\times d_z}
the context: encoder output, text-encoder states, image features, raw input bytes
WQ∈Rdx×dk, WK∈Rdz×dkW_Q\in\mathbb R^{d_x\times d_k},\ W_K\in\mathbb R^{d_z\times d_k}
map both sides into a shared dkd_k-dimensional matching space; dx≠dzd_x\ne d_z is allowed
AA
a rectangular nq×mn_q\times m matrix; row ii is the distribution of query ii over the mm context positions
MM
usually only a padding mask on the context; no causal mask, since the context is fully known

The arithmetic is exactly Scaled dot-product attention. What changes is the roles. The output length follows the queries, so the layer can turn mm context tokens into any number nqn_q of outputs: 4096 latent pixels from 77 text tokens, 100 object slots from thousands of image features, or 512 latents from 50,176 pixels. The cost is O(nq m dk)O(n_q\,m\,d_k) time and nq×mn_q\times m weights per head, linear in each length. Self-attention over the concatenation [X;Z][X;Z] would cost (nq+m)2(n_q+m)^2.

In a Transformer decoder layer the order is self-attention, then cross-attention, then the feed-forward block, each with a residual connection and layer norm. The residual matters: cross-attention writes context information into the query stream, x←x+CrossAttn(x,Z)x\leftarrow x+\mathrm{CrossAttn}(x,Z), and the model can learn to ignore it where the context is irrelevant. Flamingo makes this explicit with a gate: x←x+tanh⁡(α) CrossAttn(x,Z)x\leftarrow x+\tanh(\alpha)\,\mathrm{CrossAttn}(x,Z) with α\alpha initialised at 0, so the frozen language model starts out exactly unchanged.

Where the queries and the keys come from

modelqueries (X)keys and values (Z)notes
Transformer, T5 (encoder–decoder)decoder statesfinal encoder outputevery decoder layer attends to the same encoder output
Whispertext-token decoder statesaudio encoder output, 1500 positions for 30 sspeech recognition and translation
Latent diffusion / Stable Diffusion v1U-Net latent positions, 64×64=409664\times64=4096 at the top level for 512×512512\times512 imagesfrozen CLIP ViT-L/14 text encoder, 77 tokens × 768Q=WQφ(zt)Q=W_Q\varphi(z_t), K,VK,V from τθ(y)\tau_\theta(y) (Rombach et al. 2022)
DETR100 learned object queriesCNN feature map plus positional encodingseach query decodes one box or 'no object'
Perceiverlatent array, N=512N=512 for ImageNetinput array, M=2242=50,176M=224^2=50{,}176 pixelscost O(MN)O(MN), then self-attention among latents only
Flamingofrozen language-model tokensvisual tokens from a Perceiver Resampler (64 per image)gated: tanh⁡(α)\tanh(\alpha), α=0\alpha=0 at initialisation
Cross-attention in widely used models

Perceiver turns the usual picture around. Instead of a short query sequence reading a long context because the task requires it, it chooses a short learned latent array N≪MN\ll M so that the expensive part, attention over the input, is linear in MM. Only the small latent array then goes through deep self-attention at O(N2)O(N^2) per layer. Perceiver IO (ICLR 2022) adds a second cross-attention at the output, where task-specific output queries read the latents. Many efficient designs, such as Multimodal transformers with resampler modules and Retrieval-augmented transformers that attend to retrieved chunks, reuse this pattern of a small set of queries reading a large memory.

Q from 10×10 image latents, K and V from 7 text tokens · out = softmax(QKᵀ/√d)Voutput Σⱼ aⱼvⱼ (click a latent)map of “ball” (max 0.39)latent (4, 3) → keys<sot>0.027a0.012red0.550ball0.391on0.008green0.006grass0.006Σ = 1.000 over keysword: redmax map changevs “red”: 0.000prompt tokens (keys/values), mean weight over all 100 latent queries · click to map<sot>0.48a0.05red0.08ball0.07on0.03green0.12grass0.17scores per head: cross n_q·m = 100·7 = 700 · joint self-attn (n_q+m)² = 11,449output: one row per query → 100 vectors, the image shape; text length only sets mSD v1 at 64×64 latents, 77 CLIP tokens: cross 315,392 vs self 16,777,216 scoreskeys decide where each word lands; values decide what it writes there
colour word
map token
fig.Text-to-image cross-attention in the style of a diffusion U-Net. A 10 × 10 grid of image latents provides 100 queries. The prompt '<sot> a red ball on green grass' provides 7 keys and values. Latent features and keys are hand-set (object, ground, background), and each value writes a colour. The selected latent in the ball puts 0.550 on 'red' and 0.391 on 'ball'. A ground latent puts 0.550 on 'grass' and 0.364 on 'green'. A sky latent gives 0.826 to the start token <sot>, which acts as an attention sink and takes 0.483 of the mean weight over the grid. Switch 'red' to 'blue' or 'yellow': the ball changes colour, but the colour word's key barely moves, so no attention weight changes by more than 0.039 (blue) or 0.004 (yellow). Keys decide where a word acts and values decide what it writes, which is the observation behind Prompt-to-Prompt editing. Resize the grid from 6 × 6 to 16 × 16: the output always has one vector per latent, and the score count n_q·m grows from 252 to 1,792, against 69,169 for self-attention over all 263 latents and tokens. At Stable Diffusion v1's top resolution, 4096 latents × 77 tokens give 315,392 cross-attention scores per head, against 16,777,216 for the latents' own self-attention.

Worked example: two queries, three context tokens

Decoder queries (already projected) q1=(1,0)q_1=(1,0), q2=(0,1)q_2=(0,1). Context keys k1=(1,1)k_1=(1,1), k2=(2,0)k_2=(2,0), k3=(0,2)k_3=(0,2) and values v1=(1,0)v_1=(1,0), v2=(0,1)v_2=(0,1), v3=(1,1)v_3=(1,1), with dk=2d_k=2. In a second batch the third context position is padding.

queryscores qᵀkⱼ/√2AoutputA (k₃ padded)output (k₃ padded)
q₁ = (1, 0)0.7071, 1.4142, 00.2840, 0.5760, 0.1400(0.4240, 0.7160)0.3302, 0.6698, 0(0.3302, 0.6698)
q₂ = (0, 1)0.7071, 0, 1.41420.2840, 0.1400, 0.5760(0.8600, 0.7160)0.6698, 0.3302, 0(0.6698, 0.3302)
A 2 × 3 attention matrix, without and with a padding mask on context position 3 (checked in PyTorch)

Reading the table

  1. 1AA is 2×32\times3: two queries, three context positions. The output is 2×22\times2, one row per query, whatever mm is.
  2. 2q1q_1 matches k2k_2 best (2/2=1.41422/\sqrt2=1.4142) and q2q_2 matches k3k_3; both match k1k_1 equally. Output 1: 0.284 (1,0)+0.576 (0,1)+0.14 (1,1)=(0.4240, 0.7160)0.284\,(1,0)+0.576\,(0,1)+0.14\,(1,1)=(0.4240,\,0.7160).
  3. 3Padding k3k_3 sets its score to −∞-\infty. Row 2 renormalises over k1,k2k_1,k_2: e0.7071/(e0.7071+e0)=0.6698e^{0.7071}/(e^{0.7071}+e^{0})=0.6698.
  4. 4The keys and values depend only on ZZ. When the decoder produces one more token, only a new query row is added; KK and VV are computed once per source sentence and reused at every step.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class CrossAttention(nn.Module):
4 """Queries from x (n_q tokens, width d_model); keys and values from context (m tokens, width d_ctx)."""
5 def __init__(self, d_model, d_ctx, heads=8, d_head=64):
6 super().__init__()
7 self.h, inner = heads, heads * d_head
8 self.to_q = nn.Linear(d_model, inner, bias=False)
9 self.to_k = nn.Linear(d_ctx, inner, bias=False) # context may have a different width
10 self.to_v = nn.Linear(d_ctx, inner, bias=False)
11 self.to_out = nn.Linear(inner, d_model)
12
13 def kv(self, context):
14 # computed once per context; reused at every decoding step
15 split = lambda t: t.unflatten(-1, (self.h, -1)).transpose(1, 2) # (N, h, m, d_head)
16 return split(self.to_k(context)), split(self.to_v(context))
17
18 def forward(self, x, context=None, kv=None, context_mask=None):
19 # x (N, n_q, d_model); context (N, m, d_ctx); context_mask (N, m), True = real token
20 k, v = kv if kv is not None else self.kv(context)
21 q = self.to_q(x).unflatten(-1, (self.h, -1)).transpose(1, 2) # (N, h, n_q, d_head)
22 mask = None if context_mask is None else context_mask[:, None, None, :] # (N, 1, 1, m)
23 y = F.scaled_dot_product_attention(q, k, v, attn_mask=mask) # (N, h, n_q, d_head)
24 return self.to_out(y.transpose(1, 2).flatten(2)) # (N, n_q, d_model)
25
26torch.manual_seed(0)
27# Text-to-image, Stable Diffusion v1 sizes: 64x64 latent positions at width 320, 77 CLIP tokens of width 768
28N, n_q, m = 1, 64 * 64, 77
29x = torch.randn(N, n_q, 320)
30text = torch.randn(N, m, 768)
31attn = CrossAttention(d_model=320, d_ctx=768, heads=8, d_head=40)
32print(attn(x, text).shape) # torch.Size([1, 4096, 320]): one output per query
33
34# Encoder-decoder: a padded source, decoder queries attend to the real source tokens only
35src = torch.randn(2, 12, 512)
36src_mask = torch.arange(12)[None, :] < torch.tensor([[12], [7]]) # (2, 12)
37xattn = CrossAttention(512, 512)
38kv = xattn.kv(src) # K, V cached once: 2 x (2, 8, 12, 64)
39tgt = torch.randn(2, 5, 512)
40full = xattn(tgt, kv=kv, context_mask=src_mask) # all 5 target positions (teacher forcing)
41step = xattn(tgt[:, -1:], kv=kv, context_mask=src_mask) # one new decoding step
42print(full.shape, step.shape) # torch.Size([2, 5, 512]) torch.Size([2, 1, 512])
43print(torch.allclose(full[:, -1:], step, atol=1e-5)) # True: no causal mask needed
44
45# Changing a padded source position has no effect on sequence 1
46src2 = src.clone(); src2[1, 9] += 100.0
47print(torch.allclose(xattn(tgt, src2, context_mask=src_mask), full, atol=1e-5)) # True
A multi-head cross-attention module with a context of different width, a padding mask and cached keys and values

Use when

  • +Conditions any stream on any other: different lengths, widths and modalities
  • +Cost O(nq m)O(n_q\,m) rather than O((nq+m)2)O((n_q+m)^2); with few queries it is linear in the context length (Perceiver)
  • +Context keys and values are computed once and cached for every decoding step
  • +Attention maps give an interpretable soft alignment between the two inputs

Avoid when

  • −The context side is not updated by the layer; information flows one way per layer
  • −Every decoder layer stores its own K,VK,V of the context: memory 2 L m d2\,L\,m\,d per sequence
  • −With few queries (Perceiver latents) the bottleneck can drop input detail
  • −Mask conventions and query/key roles are easy to get backwards

Check yourself

A Stable Diffusion U-Net block cross-attends from 4096 latent positions to 77 text tokens. What are the shapes of the attention matrix per head and of the output?

Check yourself

During greedy decoding with an encoder–decoder Transformer, which parts of a cross-attention layer must be recomputed at each new target step?

Remember

  • ✓Y=softmax(XWQ(ZWK)⊤/dk) ZWVY=\mathrm{softmax}\big(XW_Q(ZW_K)^\top/\sqrt{d_k}\big)\,ZW_V: queries from XX (nqn_q rows), keys and values from the context ZZ (mm rows); A∈Rnq×mA\in\mathbb R^{n_q\times m}.
  • ✓Output has one row per query; cost O(nq m dk)O(n_q\,m\,d_k), linear in each length; only a context padding mask, never a causal one.
  • ✓Encoder–decoder Transformers and T5 cross-attend from every decoder layer to the final encoder output; K and V are cached once per source.
  • ✓Stable Diffusion: U-Net latents query 77 CLIP text tokens; Perceiver: a small latent array queries a huge input; DETR: 100 object queries read image features.
  • ✓Keys decide where a context token is read, values decide what is written, and a residual (or Flamingo's tanh gate) adds it to the query stream.
← back to Deep Learning roadmap