Definition
Cross-attention (also encoder–decoder attention) is attention in which the queries come from one sequence and the keys and values from a different sequence , often called the context or memory: . Each of the positions of reads a weighted mixture of the positions of . The output has the shape of , and only supplies content.
It is the original use of attention. In the RNN translation models of Bahdanau attention (2014) and Luong attention (2015), the decoder state was the query and the encoder states were the keys and values: cross-attention in all but name. Vaswani et al. ("Attention Is All You Need", NeurIPS 2017) kept it as one of three attention sub-layers of the Transformer. In every decoder layer, "the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder", so every target position can attend over all source positions. The other two sub-layers are encoder Self-attention and masked decoder self-attention (see Causal attention).
Because the two sides need not share a modality, length or width, the same layer became the standard way to condition one model on another. T5 (Raffel et al., JMLR 2020) and Whisper are encoder–decoders that use it to read text or audio. Latent diffusion (Rombach et al., CVPR 2022, the basis of Stable Diffusion) injects a text prompt into a U-Net by letting image positions query text-encoder states. DETR (Carion et al., ECCV 2020) decodes 100 learned object queries against CNN image features. Perceiver (Jaegle et al., ICML 2021) uses a small latent array to query a very large input. Flamingo (Alayrac et al., NeurIPS 2022) lets a frozen language model attend to visual tokens through new gated cross-attention layers.
The layer
- the stream being updated: decoder states, image latents, object queries, a latent array
- the context: encoder output, text-encoder states, image features, raw input bytes
- map both sides into a shared -dimensional matching space; is allowed
- a rectangular matrix; row is the distribution of query over the context positions
- usually only a padding mask on the context; no causal mask, since the context is fully known
The arithmetic is exactly Scaled dot-product attention. What changes is the roles. The output length follows the queries, so the layer can turn context tokens into any number of outputs: 4096 latent pixels from 77 text tokens, 100 object slots from thousands of image features, or 512 latents from 50,176 pixels. The cost is time and weights per head, linear in each length. Self-attention over the concatenation would cost .
In a Transformer decoder layer the order is self-attention, then cross-attention, then the feed-forward block, each with a residual connection and layer norm. The residual matters: cross-attention writes context information into the query stream, , and the model can learn to ignore it where the context is irrelevant. Flamingo makes this explicit with a gate: with initialised at 0, so the frozen language model starts out exactly unchanged.
Where the queries and the keys come from
| model | queries (X) | keys and values (Z) | notes |
|---|---|---|---|
| Transformer, T5 (encoder–decoder) | decoder states | final encoder output | every decoder layer attends to the same encoder output |
| Whisper | text-token decoder states | audio encoder output, 1500 positions for 30 s | speech recognition and translation |
| Latent diffusion / Stable Diffusion v1 | U-Net latent positions, at the top level for images | frozen CLIP ViT-L/14 text encoder, 77 tokens × 768 | , from (Rombach et al. 2022) |
| DETR | 100 learned object queries | CNN feature map plus positional encodings | each query decodes one box or 'no object' |
| Perceiver | latent array, for ImageNet | input array, pixels | cost , then self-attention among latents only |
| Flamingo | frozen language-model tokens | visual tokens from a Perceiver Resampler (64 per image) | gated: , at initialisation |
Perceiver turns the usual picture around. Instead of a short query sequence reading a long context because the task requires it, it chooses a short learned latent array so that the expensive part, attention over the input, is linear in . Only the small latent array then goes through deep self-attention at per layer. Perceiver IO (ICLR 2022) adds a second cross-attention at the output, where task-specific output queries read the latents. Many efficient designs, such as Multimodal transformers with resampler modules and Retrieval-augmented transformers that attend to retrieved chunks, reuse this pattern of a small set of queries reading a large memory.
Worked example: two queries, three context tokens
Decoder queries (already projected) , . Context keys , , and values , , , with . In a second batch the third context position is padding.
| query | scores qᵀkⱼ/√2 | A | output | A (k₃ padded) | output (k₃ padded) |
|---|---|---|---|---|---|
| q₁ = (1, 0) | 0.7071, 1.4142, 0 | 0.2840, 0.5760, 0.1400 | (0.4240, 0.7160) | 0.3302, 0.6698, 0 | (0.3302, 0.6698) |
| q₂ = (0, 1) | 0.7071, 0, 1.4142 | 0.2840, 0.1400, 0.5760 | (0.8600, 0.7160) | 0.6698, 0.3302, 0 | (0.6698, 0.3302) |
Reading the table
- 1 is : two queries, three context positions. The output is , one row per query, whatever is.
- 2 matches best () and matches ; both match equally. Output 1: .
- 3Padding sets its score to . Row 2 renormalises over : .
- 4The keys and values depend only on . When the decoder produces one more token, only a new query row is added; and are computed once per source sentence and reused at every step.
1import torch, torch.nn as nn, torch.nn.functional as F23class CrossAttention(nn.Module):4 """Queries from x (n_q tokens, width d_model); keys and values from context (m tokens, width d_ctx)."""5 def __init__(self, d_model, d_ctx, heads=8, d_head=64):6 super().__init__()7 self.h, inner = heads, heads * d_head8 self.to_q = nn.Linear(d_model, inner, bias=False)9 self.to_k = nn.Linear(d_ctx, inner, bias=False) # context may have a different width10 self.to_v = nn.Linear(d_ctx, inner, bias=False)11 self.to_out = nn.Linear(inner, d_model)1213 def kv(self, context):14 # computed once per context; reused at every decoding step15 split = lambda t: t.unflatten(-1, (self.h, -1)).transpose(1, 2) # (N, h, m, d_head)16 return split(self.to_k(context)), split(self.to_v(context))1718 def forward(self, x, context=None, kv=None, context_mask=None):19 # x (N, n_q, d_model); context (N, m, d_ctx); context_mask (N, m), True = real token20 k, v = kv if kv is not None else self.kv(context)21 q = self.to_q(x).unflatten(-1, (self.h, -1)).transpose(1, 2) # (N, h, n_q, d_head)22 mask = None if context_mask is None else context_mask[:, None, None, :] # (N, 1, 1, m)23 y = F.scaled_dot_product_attention(q, k, v, attn_mask=mask) # (N, h, n_q, d_head)24 return self.to_out(y.transpose(1, 2).flatten(2)) # (N, n_q, d_model)2526torch.manual_seed(0)27# Text-to-image, Stable Diffusion v1 sizes: 64x64 latent positions at width 320, 77 CLIP tokens of width 76828N, n_q, m = 1, 64 * 64, 7729x = torch.randn(N, n_q, 320)30text = torch.randn(N, m, 768)31attn = CrossAttention(d_model=320, d_ctx=768, heads=8, d_head=40)32print(attn(x, text).shape) # torch.Size([1, 4096, 320]): one output per query3334# Encoder-decoder: a padded source, decoder queries attend to the real source tokens only35src = torch.randn(2, 12, 512)36src_mask = torch.arange(12)[None, :] < torch.tensor([[12], [7]]) # (2, 12)37xattn = CrossAttention(512, 512)38kv = xattn.kv(src) # K, V cached once: 2 x (2, 8, 12, 64)39tgt = torch.randn(2, 5, 512)40full = xattn(tgt, kv=kv, context_mask=src_mask) # all 5 target positions (teacher forcing)41step = xattn(tgt[:, -1:], kv=kv, context_mask=src_mask) # one new decoding step42print(full.shape, step.shape) # torch.Size([2, 5, 512]) torch.Size([2, 1, 512])43print(torch.allclose(full[:, -1:], step, atol=1e-5)) # True: no causal mask needed4445# Changing a padded source position has no effect on sequence 146src2 = src.clone(); src2[1, 9] += 100.047print(torch.allclose(xattn(tgt, src2, context_mask=src_mask), full, atol=1e-5)) # TrueUse when
- +Conditions any stream on any other: different lengths, widths and modalities
- +Cost rather than ; with few queries it is linear in the context length (Perceiver)
- +Context keys and values are computed once and cached for every decoding step
- +Attention maps give an interpretable soft alignment between the two inputs
Avoid when
- −The context side is not updated by the layer; information flows one way per layer
- −Every decoder layer stores its own of the context: memory per sequence
- −With few queries (Perceiver latents) the bottleneck can drop input detail
- −Mask conventions and query/key roles are easy to get backwards
Check yourself
A Stable Diffusion U-Net block cross-attends from 4096 latent positions to 77 text tokens. What are the shapes of the attention matrix per head and of the output?
Check yourself
During greedy decoding with an encoder–decoder Transformer, which parts of a cross-attention layer must be recomputed at each new target step?
Remember
- ✓: queries from ( rows), keys and values from the context ( rows); .
- ✓Output has one row per query; cost , linear in each length; only a context padding mask, never a causal one.
- ✓Encoder–decoder Transformers and T5 cross-attend from every decoder layer to the final encoder output; K and V are cached once per source.
- ✓Stable Diffusion: U-Net latents query 77 CLIP text tokens; Perceiver: a small latent array queries a huge input; DETR: 100 object queries read image features.
- ✓Keys decide where a context token is read, values decide what is written, and a residual (or Flamingo's tanh gate) adds it to the query stream.