Multimodal transformers

Turn every modality into a sequence of vectors (image patches, audio frames, text tokens), then decide where they meet: one joint encoder, co-attention between two streams, only a contrastive loss (CLIP), cross-attention into a frozen LM (Flamingo), or projected tokens fed to the LM (LLaVA).

15 min read1366 wordscore0 viewsupdated 2026-10-12

Definition

Multimodal transformer a Transformer that processes more than one modality, such as images and text, audio and text, or video. Each modality is first tokenised into a sequence of dd-dimensional vectors by a modality-specific embedder (patch projection, spectrogram frames, word embeddings). The architecture then fixes where the modalities interact: in every layer through joint Self-attention, in some layers through Cross-attention, or only at the output through a similarity score. Most designs reuse an existing pre-trained encoder or language model for at least one side.

Attention does not care where its vectors came from. Once an image is a sequence of patch tokens (see Vision Transformers (ViT)) and a sentence a sequence of word tokens, the same layers can relate them. The research question since 2019 has been which connection pattern to use and what to pre-train. Early vision-and-language models (ViLBERT, LXMERT, VisualBERT, UNITER, 2019–2020) used region features from an object detector and BERT-style objectives. CLIP and ALIGN (2021) replaced labels with hundreds of millions of web image–text pairs and a contrastive loss. Flamingo (2022) and LLaVA (2023) connected vision encoders to large language models, so a single model can answer free-form questions about images.

Tokenising a modality

Images: cut into P×PP\times P patches and project each, N=HW/P2N=HW/P^2 tokens. A 2242224^2 image with P=14P=14 gives 256 tokens; at 3362336^2 it gives 576. Earlier models instead took 10–100 region features from a Faster R-CNN detector. Video: 3-D "tubelets" of t×P×Pt\times P\times P pixels (ViViT, Arnab et al. 2021), or per-frame tokens pooled over time. Audio: a log-Mel spectrogram, either cut into 2-D patches (AST, Gong et al. 2021) or treated as a sequence of frames. Whisper (Radford et al., ICML 2023) uses 80 Mel channels with a 25 ms window and 10 ms hop. A 30 s clip gives 3000 frames, and two 1-D convolutions (the second with stride 2) turn them into 1500 tokens. Discrete tokens: a VQ-style tokenizer maps an image to codebook indices that share a vocabulary with text. Chameleon (Meta, 2024) encodes a 5122512^2 image as 1024 tokens from an 8192-entry codebook and trains one model on mixed sequences.

Each modality also needs position information in its own geometry: 2-D position embeddings for patches, time for audio, both for video. It also needs type information, a learned modality embedding or special tokens such as <image> and </image>, so the model can tell which tokens came from where. The tokenizers set the sequence lengths, and with them most of the cost. An image is often worth hundreds of tokens, more than the text that comes with it.

Five ways to fuse

patternwhere modalities meetexamplesgood for
single-stream (early fusion)self-attention over the concatenation [image; text] in every layerVisualBERT, UNITER, ViLT (2019–21); Chameleon with discrete tokensfine-grained reasoning (VQA, grounding)
dual-stream co-attentionco-attention layers: each stream's queries attend to the other stream's keys and valuesViLBERT, LXMERT (2019)keeping separate capacity per modality
dual encoder (late fusion)only a dot product of pooled embeddings, trained contrastivelyCLIP, ALIGN (2021)retrieval, zero-shot classification, cheap indexing
cross-attention into an LMnew gated cross-attention layers inside a frozen LM; text queries, visual keys/valuesFlamingo (2022)few-shot, interleaved image–text
projection into an LMvisual features mapped by a small network into the LM's input embedding spaceLLaVA (2023), BLIP-2's Q-Former (2023)instruction following, chat about images
Fusion patterns (a model can mix them)

Co-attention (ViLBERT)

H~V=HV+softmax ⁣((HVWVQ)(HLWLK)⊤dk)HLWLV,H~L=HL+softmax ⁣((HLWLQ)(HVWVK)⊤dk)HVWVV\tilde H_V=H_V+\mathrm{softmax}\!\Big(\frac{(H_VW^Q_V)(H_LW^K_L)^{\top}}{\sqrt{d_k}}\Big)H_LW^V_L,\qquad \tilde H_L=H_L+\mathrm{softmax}\!\Big(\frac{(H_LW^Q_L)(H_VW^K_V)^{\top}}{\sqrt{d_k}}\Big)H_VW^V_V
HV, HLH_V,\ H_L
hidden states of the visual stream (region features) and the linguistic stream (word pieces)
WVQ, WLK,…W^Q_V,\ W^K_L,\dots
each stream has its own projections; the queries come from one stream and the keys and values from the other
H~\tilde H
after the co-attention sub-layer each stream runs its own self-attention and FFN (shown with Multi-head attention in practice)

Lu, Batra, Parikh and Lee, "ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks" (NeurIPS 2019), ran two BERT-like streams and exchanged information only in co-attention layers. Image features were Faster R-CNN region features. Pre-training on Conceptual Captions (about 3.3M image–caption pairs) used masked multimodal modelling and image–text alignment prediction. In a single-stream model the image and text keys share one softmax and compete for attention mass. In co-attention every query gets one distribution over its own modality and a separate one over the other.

Contrastive dual encoders (CLIP)

sij=ui⊤vjτ,L=12B∑i=1B[−log⁡esii∑jesij−log⁡esii∑jesji]s_{ij}=\frac{u_i^{\top}v_j}{\tau},\qquad \mathcal L=\frac1{2B}\sum_{i=1}^{B}\Big[-\log\frac{e^{s_{ii}}}{\sum_{j}e^{s_{ij}}}-\log\frac{e^{s_{ii}}}{\sum_{j}e^{s_{ji}}}\Big]
ui, vju_i,\ v_j
L2-normalised embeddings of image ii and text jj from two separate encoders plus linear projections
τ\tau
learned temperature, initialised to 0.07, with the logit scale 1/τ1/\tau clipped at 100
BB
batch of matched pairs; the other B−1B-1 captions are the negatives for each image and vice versa (32,768 in CLIP)

Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (ICML 2021), trained CLIP on 400M web image–text pairs with this symmetric InfoNCE loss. The two encoders never attend to each other. The text encoder is a causal Transformer read out at its end-of-text token, and the image encoder a ResNet or ViT. Classification becomes retrieval: embed one prompt per class ("a photo of a {label}") and pick the class with the most similar text. The ViT-L/14 model at 336 px reached 76.2% zero-shot ImageNet accuracy, matching the original supervised ResNet-50 without using any of its 1.28M labelled images. Because embeddings are computed independently, a whole image collection can be indexed once. Fine-grained reasoning ("is the cup left of the plate?") is weak, because the only interaction is one dot product.

Cross-attention into a frozen LM (Flamingo)

y←y+tanh⁡(αxattn)⋅XAttn(q=y, kv=Z),y←y+tanh⁡(αff)⋅FFW(y),Z=Resampler(Xv)∈R64×dy\leftarrow y+\tanh(\alpha_{\text{xattn}})\cdot\mathrm{XAttn}\big(q=y,\ kv=Z\big),\qquad y\leftarrow y+\tanh(\alpha_{\text{ff}})\cdot\mathrm{FFW}(y),\qquad Z=\mathrm{Resampler}(X_v)\in\mathbb R^{64\times d}
yy
text hidden states entering a frozen LM layer
XvX_v
features from a frozen vision encoder (an NFNet-F6 pre-trained contrastively); their number depends on image size and video length
Resampler\mathrm{Resampler}
Perceiver resampler: 64 learned latent queries cross-attend to XvX_v and output a fixed 64 visual tokens
α\alpha
learnable scalars initialised to 0, so tanh⁡(α)=0\tanh(\alpha)=0 and the model starts exactly as the frozen LM

Alayrac et al., "Flamingo: a Visual Language Model for Few-Shot Learning" (NeurIPS 2022), kept a pre-trained Chinchilla LM (70B for Flamingo-80B) and the vision encoder frozen, and trained only the resampler and the inserted GATED XATTN-DENSE layers. The tanh gates matter: random cross-attention added to a frozen LM at full strength would destroy its predictions at step 0. Starting from zero, the gates let the visual signal grow as it becomes useful. Inputs are interleaved web documents (M3W, 43M pages), image–text and video–text pairs. A per-image mask lets each text token cross-attend only to the visual tokens of the most recent preceding image. The LM self-attention still sees all previous text, so the model can use in-context examples. Flamingo was evaluated on 16 image and video benchmarks and set a new few-shot state of the art on the 9 that had published few-shot results. On 6 of them it beat fine-tuned models while using only 32 task-specific examples.

Projection into the LM (LLaVA)

Hv=W⋅Zv,Zv=g(Xv)∈RN×1024,LM input=[Hv; Embed(prompt)]H_v=W\cdot Z_v,\qquad Z_v=g(X_v)\in\mathbb R^{N\times1024},\qquad \text{LM input}=\big[H_v;\ \mathrm{Embed}(\text{prompt})\big]
gg
a frozen CLIP ViT-L/14 image encoder; ZvZ_v are its patch ("grid") features
WW
a linear map to the LM width in LLaVA; a 2-layer MLP with GELU in LLaVA-1.5
NN
(224/14)2=256(224/14)^2=256 in LLaVA, (336/14)2=576(336/14)^2=576 in LLaVA-1.5

Liu, Li, Wu and Lee, "Visual Instruction Tuning" (NeurIPS 2023), showed that the simplest connector works if the training data is right. Stage 1 trains only WW on 595K filtered CC3M image–caption pairs, aligning visual features with the word-embedding space. Stage 2 fine-tunes WW and the LM (Vicuna) on 158K GPT-4-generated instruction-following conversations about images. The projected patches are ordinary input tokens to a Decoder-only transformer, so the causal mask covers them as well: text tokens attend to every image token, and image tokens never attend to text. LLaVA-1.5 ("Improved Baselines with Visual Instruction Tuning", CVPR 2024) used an MLP connector and 336-px input, and became a common baseline. BLIP-2 (Li et al., ICML 2023) sits between Flamingo and LLaVA. Its Q-Former compresses the image into 32 learned query tokens, which are then projected into a frozen LM's input.

rows = queries, columns = keys · ■ same modality ■ cross-modal ■ masked6 image patch tokens p, then 7 caption tokensp0p1p2p3p4p5adognapsonaredsofap0p1p2p3p4p5adognapsonaredsofaquery “naps” sees 6 visual + 3 text keys · click a rowtoy mask (13 × 13 = 169):same-modality 49cross-modal 42masked 78at scale (S = 336, P = 14, T = 32):visual tokens 576joint sequence 608scores / layer 185,136cross-modal scores 18,432patch features → projection W → word-embedding space; one causal mask covers both,so text attends to the image, but image tokens never see the textLLaVA-1.5 at 336²: (336/14)² = 576 image tokens precede the prompt
fusion
fig.The attention mask over one toy image and the caption 'a dog naps on a red sofa'. Blue entries are same-modality, orange cross-modal, grey masked. With 6 patches the LLaVA mask is causal over all 13 tokens: 49 same-modality and 42 cross-modal entries, and 78 masked, including every image-to-text entry. Single-stream fills the whole square. ViLBERT splits it into a co-attention sub-layer (only the off-diagonal blocks) and a self-attention sub-layer (only the diagonal blocks). CLIP has no orange at all. In Flamingo mode the patches are replaced by 3 resampler latents that only serve as keys and values, and the gate slider fades them by tanh(α). The right panel counts the same patterns at real scale with 14-px patches and 32 text tokens. At 336² LLaVA puts 576 image tokens before the text, 185,136 causal scores per layer, of which 18,432 are text-to-image. Single-stream needs 369,664. Flamingo's LM sees 64 latents at any resolution: 2,576 scores, 2,048 of them cross-attention. Moving to 672² multiplies LLaVA's image tokens by 4, to 2,304, and its scores to 2,729,616.

Worked example: CLIP loss for a batch of three

imagecaption 1caption 2caption 3row softmax (image → text)
image 10.300.100.050.9212, 0.0529, 0.0259
image 20.120.280.200.0716, 0.7039, 0.2245
image 30.020.150.250.0293, 0.1877, 0.7830
Cosine similarities uᵢᵀvⱼ (rows images, columns captions; diagonal = matching pairs) and the softmax rows at τ = 0.07 (computed in Node)

From similarities to the symmetric loss

  1. 1Divide by τ=0.07\tau=0.07: image 2's logits are (1.714, 4.000, 2.857)(1.714,\ 4.000,\ 2.857). Similarities that differ by only 0.08 (0.28 vs 0.20) become a probability of 0.7039 against 0.2245. The small temperature makes the loss sensitive to small cosine gaps.
  2. 2Image→text cross-entropies −log⁡-\log of the diagonal: 0.0821, 0.3511, 0.24460.0821,\ 0.3511,\ 0.2446, mean 0.22590.2259.
  3. 3Text→image uses the columns. For caption 3 the column softmax is (0.0167, 0.1267, 0.6464)(0.0167,\ 0.1267,\ 0.6464), so its loss is 0.43630.4363. The three column losses are 0.0905, 0.2091, 0.43630.0905,\ 0.2091,\ 0.4363, mean 0.24530.2453.
  4. 4L=(0.2259+0.2453)/2=\mathcal L=(0.2259+0.2453)/2= 0.2356. A random encoder would give about log⁡3=1.0986\log3=1.0986; with CLIP's batch of 32,768 the chance level is log⁡32768=10.4\log32768=10.4.
  5. 5Token budget for LLaVA-1.5: 336/14=24336/14=24, 242=24^2= 576 visual tokens per image. Its MLP connector for the 7B LM (1024→4096→40961024\to4096\to4096) has 1024⋅4096+4096+4096⋅4096+4096=20,979,7121024\cdot4096+4096+4096\cdot4096+4096=20{,}979{,}712 parameters, about 0.3% of the LM.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3# 1) CLIP: symmetric contrastive loss over B matched pairs
4def clip_loss(img_emb, txt_emb, logit_scale):
5 u = F.normalize(img_emb, dim=-1) # (B, D)
6 v = F.normalize(txt_emb, dim=-1) # (B, D)
7 logits = logit_scale * u @ v.t() # (B, B); logit_scale = 1/tau
8 target = torch.arange(len(u))
9 return (F.cross_entropy(logits, target) + F.cross_entropy(logits.t(), target)) / 2
10
11print(clip_loss(torch.randn(4, 512), torch.randn(4, 512), 1 / 0.07).shape) # torch.Size([])
12
13# 2) LLaVA-1.5: MLP projector from CLIP ViT-L/14-336 patch features to the LM width
14proj = nn.Sequential(nn.Linear(1024, 4096), nn.GELU(), nn.Linear(4096, 4096))
15patches = torch.randn(1, 576, 1024) # (336/14)^2 = 576 patch features
16txt = torch.randn(1, 32, 4096) # embedded prompt tokens
17llm_input = torch.cat([proj(patches), txt], dim=1)
18print(llm_input.shape) # torch.Size([1, 608, 4096])
19print(sum(p.numel() for p in proj.parameters())) # 20979712
20
21# 3) Flamingo-style gated cross-attention, inserted before a frozen LM layer
22class GatedXAttn(nn.Module):
23 def __init__(self, d, heads):
24 super().__init__()
25 self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
26 self.xattn = nn.MultiheadAttention(d, heads, batch_first=True)
27 self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
28 self.a_attn = nn.Parameter(torch.zeros(1)) # tanh(0) = 0: identity at init
29 self.a_ff = nn.Parameter(torch.zeros(1))
30
31 def forward(self, y, z): # y: (B, T, d) text, z: (B, 64, d) latents
32 y = y + self.a_attn.tanh() * self.xattn(self.ln1(y), z, z, need_weights=False)[0]
33 return y + self.a_ff.tanh() * self.ff(self.ln2(y))
34
35blk = GatedXAttn(512, 8)
36y = torch.randn(2, 32, 512)
37out = blk(y, torch.randn(2, 64, 512))
38print(out.shape, torch.equal(out, y)) # torch.Size([2, 32, 512]) True
Three connectors in PyTorch: CLIP loss, a LLaVA-1.5 projector, a Flamingo gated cross-attention block

Use when

  • +One architecture for all modalities once each is tokenised; attention learns the cross-modal alignment
  • +Dual encoders: precomputable embeddings, zero-shot transfer, cheap retrieval
  • +Cross-attention and projection reuse frozen LLMs and vision encoders, so training is cheap
  • +Single-stream and LM-based fusion support fine-grained reasoning about image regions and text

Avoid when

  • −Visual tokens make sequences long; cost grows quadratically with resolution and number of images
  • −Dual encoders cannot model fine-grained relations; joint models cannot index the two sides separately
  • −Performance depends heavily on the size and quality of paired data (web alt-text is noisy)
  • −Language priors can override the image: models answer from text statistics (hallucination)

Check yourself

Which design lets you embed a billion images once and then search them with any new text query at the cost of a dot product per image?

Check yourself

Flamingo initialises the gate α of each new cross-attention layer to 0. What is the effect?

Remember

  • ✓Tokenise each modality: image patches N=(H/P)2N=(H/P)^2 (576 at 3362336^2, P=14P=14), audio log-Mel frames (Whisper: 30 s → 1500 tokens), then add position and modality information.
  • ✓Fusion: single-stream joint attention; co-attention (ViLBERT); dual encoder with CLIP loss sij=ui⊤vj/τs_{ij}=u_i^\top v_j/\tau; gated cross-attention y+tanh⁡(α) XAttn(y,Z)y+\tanh(\alpha)\,\mathrm{XAttn}(y,Z) (Flamingo); projection Hv=WZvH_v=WZ_v into the LM (LLaVA).
  • ✓CLIP: 400M pairs, symmetric InfoNCE, τ initialised 0.07, 76.2% zero-shot ImageNet with ViT-L/14@336.
  • ✓Flamingo: frozen vision encoder and LM, Perceiver resampler to 64 tokens, zero-initialised tanh gates.
  • ✓LLaVA: CLIP ViT-L/14 patch features → linear (MLP in 1.5) → LM tokens; two-stage training on 595K captions then 158K instructions.
← back to Deep Learning roadmap