Definition
Multimodal transformer a Transformer that processes more than one modality, such as images and text, audio and text, or video. Each modality is first tokenised into a sequence of -dimensional vectors by a modality-specific embedder (patch projection, spectrogram frames, word embeddings). The architecture then fixes where the modalities interact: in every layer through joint Self-attention, in some layers through Cross-attention, or only at the output through a similarity score. Most designs reuse an existing pre-trained encoder or language model for at least one side.
Attention does not care where its vectors came from. Once an image is a sequence of patch tokens (see Vision Transformers (ViT)) and a sentence a sequence of word tokens, the same layers can relate them. The research question since 2019 has been which connection pattern to use and what to pre-train. Early vision-and-language models (ViLBERT, LXMERT, VisualBERT, UNITER, 2019–2020) used region features from an object detector and BERT-style objectives. CLIP and ALIGN (2021) replaced labels with hundreds of millions of web image–text pairs and a contrastive loss. Flamingo (2022) and LLaVA (2023) connected vision encoders to large language models, so a single model can answer free-form questions about images.
Tokenising a modality
Images: cut into patches and project each, tokens. A image with gives 256 tokens; at it gives 576. Earlier models instead took 10–100 region features from a Faster R-CNN detector. Video: 3-D "tubelets" of pixels (ViViT, Arnab et al. 2021), or per-frame tokens pooled over time. Audio: a log-Mel spectrogram, either cut into 2-D patches (AST, Gong et al. 2021) or treated as a sequence of frames. Whisper (Radford et al., ICML 2023) uses 80 Mel channels with a 25 ms window and 10 ms hop. A 30 s clip gives 3000 frames, and two 1-D convolutions (the second with stride 2) turn them into 1500 tokens. Discrete tokens: a VQ-style tokenizer maps an image to codebook indices that share a vocabulary with text. Chameleon (Meta, 2024) encodes a image as 1024 tokens from an 8192-entry codebook and trains one model on mixed sequences.
Each modality also needs position information in its own geometry: 2-D position embeddings for patches, time for audio, both for video. It also needs type information, a learned modality embedding or special tokens such as <image> and </image>, so the model can tell which tokens came from where. The tokenizers set the sequence lengths, and with them most of the cost. An image is often worth hundreds of tokens, more than the text that comes with it.
Five ways to fuse
| pattern | where modalities meet | examples | good for |
|---|---|---|---|
| single-stream (early fusion) | self-attention over the concatenation [image; text] in every layer | VisualBERT, UNITER, ViLT (2019–21); Chameleon with discrete tokens | fine-grained reasoning (VQA, grounding) |
| dual-stream co-attention | co-attention layers: each stream's queries attend to the other stream's keys and values | ViLBERT, LXMERT (2019) | keeping separate capacity per modality |
| dual encoder (late fusion) | only a dot product of pooled embeddings, trained contrastively | CLIP, ALIGN (2021) | retrieval, zero-shot classification, cheap indexing |
| cross-attention into an LM | new gated cross-attention layers inside a frozen LM; text queries, visual keys/values | Flamingo (2022) | few-shot, interleaved image–text |
| projection into an LM | visual features mapped by a small network into the LM's input embedding space | LLaVA (2023), BLIP-2's Q-Former (2023) | instruction following, chat about images |
Co-attention (ViLBERT)
- hidden states of the visual stream (region features) and the linguistic stream (word pieces)
- each stream has its own projections; the queries come from one stream and the keys and values from the other
- after the co-attention sub-layer each stream runs its own self-attention and FFN (shown with Multi-head attention in practice)
Lu, Batra, Parikh and Lee, "ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks" (NeurIPS 2019), ran two BERT-like streams and exchanged information only in co-attention layers. Image features were Faster R-CNN region features. Pre-training on Conceptual Captions (about 3.3M image–caption pairs) used masked multimodal modelling and image–text alignment prediction. In a single-stream model the image and text keys share one softmax and compete for attention mass. In co-attention every query gets one distribution over its own modality and a separate one over the other.
Contrastive dual encoders (CLIP)
- L2-normalised embeddings of image and text from two separate encoders plus linear projections
- learned temperature, initialised to 0.07, with the logit scale clipped at 100
- batch of matched pairs; the other captions are the negatives for each image and vice versa (32,768 in CLIP)
Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (ICML 2021), trained CLIP on 400M web image–text pairs with this symmetric InfoNCE loss. The two encoders never attend to each other. The text encoder is a causal Transformer read out at its end-of-text token, and the image encoder a ResNet or ViT. Classification becomes retrieval: embed one prompt per class ("a photo of a {label}") and pick the class with the most similar text. The ViT-L/14 model at 336 px reached 76.2% zero-shot ImageNet accuracy, matching the original supervised ResNet-50 without using any of its 1.28M labelled images. Because embeddings are computed independently, a whole image collection can be indexed once. Fine-grained reasoning ("is the cup left of the plate?") is weak, because the only interaction is one dot product.
Cross-attention into a frozen LM (Flamingo)
- text hidden states entering a frozen LM layer
- features from a frozen vision encoder (an NFNet-F6 pre-trained contrastively); their number depends on image size and video length
- Perceiver resampler: 64 learned latent queries cross-attend to and output a fixed 64 visual tokens
- learnable scalars initialised to 0, so and the model starts exactly as the frozen LM
Alayrac et al., "Flamingo: a Visual Language Model for Few-Shot Learning" (NeurIPS 2022), kept a pre-trained Chinchilla LM (70B for Flamingo-80B) and the vision encoder frozen, and trained only the resampler and the inserted GATED XATTN-DENSE layers. The tanh gates matter: random cross-attention added to a frozen LM at full strength would destroy its predictions at step 0. Starting from zero, the gates let the visual signal grow as it becomes useful. Inputs are interleaved web documents (M3W, 43M pages), image–text and video–text pairs. A per-image mask lets each text token cross-attend only to the visual tokens of the most recent preceding image. The LM self-attention still sees all previous text, so the model can use in-context examples. Flamingo was evaluated on 16 image and video benchmarks and set a new few-shot state of the art on the 9 that had published few-shot results. On 6 of them it beat fine-tuned models while using only 32 task-specific examples.
Projection into the LM (LLaVA)
- a frozen CLIP ViT-L/14 image encoder; are its patch ("grid") features
- a linear map to the LM width in LLaVA; a 2-layer MLP with GELU in LLaVA-1.5
- in LLaVA, in LLaVA-1.5
Liu, Li, Wu and Lee, "Visual Instruction Tuning" (NeurIPS 2023), showed that the simplest connector works if the training data is right. Stage 1 trains only on 595K filtered CC3M image–caption pairs, aligning visual features with the word-embedding space. Stage 2 fine-tunes and the LM (Vicuna) on 158K GPT-4-generated instruction-following conversations about images. The projected patches are ordinary input tokens to a Decoder-only transformer, so the causal mask covers them as well: text tokens attend to every image token, and image tokens never attend to text. LLaVA-1.5 ("Improved Baselines with Visual Instruction Tuning", CVPR 2024) used an MLP connector and 336-px input, and became a common baseline. BLIP-2 (Li et al., ICML 2023) sits between Flamingo and LLaVA. Its Q-Former compresses the image into 32 learned query tokens, which are then projected into a frozen LM's input.
Worked example: CLIP loss for a batch of three
| image | caption 1 | caption 2 | caption 3 | row softmax (image → text) |
|---|---|---|---|---|
| image 1 | 0.30 | 0.10 | 0.05 | 0.9212, 0.0529, 0.0259 |
| image 2 | 0.12 | 0.28 | 0.20 | 0.0716, 0.7039, 0.2245 |
| image 3 | 0.02 | 0.15 | 0.25 | 0.0293, 0.1877, 0.7830 |
From similarities to the symmetric loss
- 1Divide by : image 2's logits are . Similarities that differ by only 0.08 (0.28 vs 0.20) become a probability of 0.7039 against 0.2245. The small temperature makes the loss sensitive to small cosine gaps.
- 2Image→text cross-entropies of the diagonal: , mean .
- 3Text→image uses the columns. For caption 3 the column softmax is , so its loss is . The three column losses are , mean .
- 4 0.2356. A random encoder would give about ; with CLIP's batch of 32,768 the chance level is .
- 5Token budget for LLaVA-1.5: , 576 visual tokens per image. Its MLP connector for the 7B LM () has parameters, about 0.3% of the LM.
1import torch, torch.nn as nn, torch.nn.functional as F23# 1) CLIP: symmetric contrastive loss over B matched pairs4def clip_loss(img_emb, txt_emb, logit_scale):5 u = F.normalize(img_emb, dim=-1) # (B, D)6 v = F.normalize(txt_emb, dim=-1) # (B, D)7 logits = logit_scale * u @ v.t() # (B, B); logit_scale = 1/tau8 target = torch.arange(len(u))9 return (F.cross_entropy(logits, target) + F.cross_entropy(logits.t(), target)) / 21011print(clip_loss(torch.randn(4, 512), torch.randn(4, 512), 1 / 0.07).shape) # torch.Size([])1213# 2) LLaVA-1.5: MLP projector from CLIP ViT-L/14-336 patch features to the LM width14proj = nn.Sequential(nn.Linear(1024, 4096), nn.GELU(), nn.Linear(4096, 4096))15patches = torch.randn(1, 576, 1024) # (336/14)^2 = 576 patch features16txt = torch.randn(1, 32, 4096) # embedded prompt tokens17llm_input = torch.cat([proj(patches), txt], dim=1)18print(llm_input.shape) # torch.Size([1, 608, 4096])19print(sum(p.numel() for p in proj.parameters())) # 209797122021# 3) Flamingo-style gated cross-attention, inserted before a frozen LM layer22class GatedXAttn(nn.Module):23 def __init__(self, d, heads):24 super().__init__()25 self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)26 self.xattn = nn.MultiheadAttention(d, heads, batch_first=True)27 self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))28 self.a_attn = nn.Parameter(torch.zeros(1)) # tanh(0) = 0: identity at init29 self.a_ff = nn.Parameter(torch.zeros(1))3031 def forward(self, y, z): # y: (B, T, d) text, z: (B, 64, d) latents32 y = y + self.a_attn.tanh() * self.xattn(self.ln1(y), z, z, need_weights=False)[0]33 return y + self.a_ff.tanh() * self.ff(self.ln2(y))3435blk = GatedXAttn(512, 8)36y = torch.randn(2, 32, 512)37out = blk(y, torch.randn(2, 64, 512))38print(out.shape, torch.equal(out, y)) # torch.Size([2, 32, 512]) TrueUse when
- +One architecture for all modalities once each is tokenised; attention learns the cross-modal alignment
- +Dual encoders: precomputable embeddings, zero-shot transfer, cheap retrieval
- +Cross-attention and projection reuse frozen LLMs and vision encoders, so training is cheap
- +Single-stream and LM-based fusion support fine-grained reasoning about image regions and text
Avoid when
- −Visual tokens make sequences long; cost grows quadratically with resolution and number of images
- −Dual encoders cannot model fine-grained relations; joint models cannot index the two sides separately
- −Performance depends heavily on the size and quality of paired data (web alt-text is noisy)
- −Language priors can override the image: models answer from text statistics (hallucination)
Check yourself
Which design lets you embed a billion images once and then search them with any new text query at the cost of a dot product per image?
Check yourself
Flamingo initialises the gate α of each new cross-attention layer to 0. What is the effect?
Remember
- ✓Tokenise each modality: image patches (576 at , ), audio log-Mel frames (Whisper: 30 s → 1500 tokens), then add position and modality information.
- ✓Fusion: single-stream joint attention; co-attention (ViLBERT); dual encoder with CLIP loss ; gated cross-attention (Flamingo); projection into the LM (LLaVA).
- ✓CLIP: 400M pairs, symmetric InfoNCE, τ initialised 0.07, 76.2% zero-shot ImageNet with ViT-L/14@336.
- ✓Flamingo: frozen vision encoder and LM, Perceiver resampler to 64 tokens, zero-initialised tanh gates.
- ✓LLaVA: CLIP ViT-L/14 patch features → linear (MLP in 1.5) → LM tokens; two-stage training on 595K captions then 158K instructions.