# Multimodal transformers

> Turn every modality into a sequence of vectors (image patches, audio frames, text tokens), then decide where they meet: one joint encoder, co-attention between two streams, only a contrastive loss (CLIP), cross-attention into a frozen LM (Flamingo), or projected tokens fed to the LM (LLaVA).

Source: https://www.mlfoundry.in/learn/multimodal-transformers  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 15 min read · updated 2026-10-12

**Multimodal transformer:** a Transformer that processes more than one modality, such as images and text, audio and text, or video. Each modality is first **tokenised** into a sequence of $d$-dimensional vectors by a modality-specific embedder (patch projection, spectrogram frames, word embeddings). The architecture then fixes **where the modalities interact**: in every layer through joint **Self-attention**, in some layers through **Cross-attention**, or only at the output through a similarity score. Most designs reuse an existing pre-trained encoder or language model for at least one side.

Attention does not care where its vectors came from. Once an image is a sequence of patch tokens (see **Vision Transformers (ViT)**) and a sentence a sequence of word tokens, the same layers can relate them. The research question since 2019 has been which connection pattern to use and what to pre-train. Early vision-and-language models (ViLBERT, LXMERT, VisualBERT, UNITER, 2019–2020) used region features from an object detector and BERT-style objectives. CLIP and ALIGN (2021) replaced labels with hundreds of millions of web image–text pairs and a contrastive loss. Flamingo (2022) and LLaVA (2023) connected vision encoders to large language models, so a single model can answer free-form questions about images.

## Tokenising a modality

**Images**: cut into $P\times P$ patches and project each, $N=HW/P^2$ tokens. A $224^2$ image with $P=14$ gives 256 tokens; at $336^2$ it gives 576. Earlier models instead took 10–100 region features from a Faster R-CNN detector. **Video**: 3-D "tubelets" of $t\times P\times P$ pixels (ViViT, Arnab et al. 2021), or per-frame tokens pooled over time. **Audio**: a log-Mel spectrogram, either cut into 2-D patches (AST, Gong et al. 2021) or treated as a sequence of frames. Whisper (Radford et al., ICML 2023) uses 80 Mel channels with a 25 ms window and 10 ms hop. A 30 s clip gives 3000 frames, and two 1-D convolutions (the second with stride 2) turn them into 1500 tokens. **Discrete tokens**: a VQ-style tokenizer maps an image to codebook indices that share a vocabulary with text. Chameleon (Meta, 2024) encodes a $512^2$ image as 1024 tokens from an 8192-entry codebook and trains one model on mixed sequences.

Each modality also needs **position** information in its own geometry: 2-D position embeddings for patches, time for audio, both for video. It also needs **type** information, a learned modality embedding or special tokens such as `<image>` and `</image>`, so the model can tell which tokens came from where. The tokenizers set the sequence lengths, and with them most of the cost. An image is often worth hundreds of tokens, more than the text that comes with it.

## Five ways to fuse

Fusion patterns (a model can mix them)

| pattern | where modalities meet | examples | good for |
| --- | --- | --- | --- |
| single-stream (early fusion) | self-attention over the concatenation [image; text] in every layer | VisualBERT, UNITER, ViLT (2019–21); Chameleon with discrete tokens | fine-grained reasoning (VQA, grounding) |
| dual-stream co-attention | co-attention layers: each stream's queries attend to the other stream's keys and values | ViLBERT, LXMERT (2019) | keeping separate capacity per modality |
| dual encoder (late fusion) | only a dot product of pooled embeddings, trained contrastively | CLIP, ALIGN (2021) | retrieval, zero-shot classification, cheap indexing |
| cross-attention into an LM | new gated cross-attention layers inside a frozen LM; text queries, visual keys/values | Flamingo (2022) | few-shot, interleaved image–text |
| projection into an LM | visual features mapped by a small network into the LM's input embedding space | LLaVA (2023), BLIP-2's Q-Former (2023) | instruction following, chat about images |

## Co-attention (ViLBERT)

$$
\tilde H_V=H_V+\mathrm{softmax}\!\Big(\frac{(H_VW^Q_V)(H_LW^K_L)^{\top}}{\sqrt{d_k}}\Big)H_LW^V_L,\qquad \tilde H_L=H_L+\mathrm{softmax}\!\Big(\frac{(H_LW^Q_L)(H_VW^K_V)^{\top}}{\sqrt{d_k}}\Big)H_VW^V_V
$$

- $H_V,\ H_L$: hidden states of the visual stream (region features) and the linguistic stream (word pieces)
- $W^Q_V,\ W^K_L,\dots$: each stream has its own projections; the queries come from one stream and the keys and values from the other
- $\tilde H$: after the co-attention sub-layer each stream runs its own self-attention and FFN (shown with **Multi-head attention** in practice)

Lu, Batra, Parikh and Lee, "ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks" (NeurIPS 2019), ran two BERT-like streams and exchanged information only in co-attention layers. Image features were Faster R-CNN region features. Pre-training on Conceptual Captions (about 3.3M image–caption pairs) used masked multimodal modelling and image–text alignment prediction. In a single-stream model the image and text keys share one softmax and compete for attention mass. In co-attention every query gets one distribution over its own modality and a separate one over the other.

## Contrastive dual encoders (CLIP)

$$
s_{ij}=\frac{u_i^{\top}v_j}{\tau},\qquad \mathcal L=\frac1{2B}\sum_{i=1}^{B}\Big[-\log\frac{e^{s_{ii}}}{\sum_{j}e^{s_{ij}}}-\log\frac{e^{s_{ii}}}{\sum_{j}e^{s_{ji}}}\Big]
$$

- $u_i,\ v_j$: L2-normalised embeddings of image $i$ and text $j$ from two separate encoders plus linear projections
- $\tau$: learned temperature, initialised to 0.07, with the logit scale $1/\tau$ clipped at 100
- $B$: batch of matched pairs; the other $B-1$ captions are the negatives for each image and vice versa (32,768 in CLIP)

Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (ICML 2021), trained CLIP on 400M web image–text pairs with this symmetric **InfoNCE loss**. The two encoders never attend to each other. The text encoder is a causal Transformer read out at its end-of-text token, and the image encoder a ResNet or ViT. Classification becomes retrieval: embed one prompt per class ("a photo of a {label}") and pick the class with the most similar text. The ViT-L/14 model at 336 px reached 76.2% zero-shot ImageNet accuracy, matching the original supervised ResNet-50 without using any of its 1.28M labelled images. Because embeddings are computed independently, a whole image collection can be indexed once. Fine-grained reasoning ("is the cup left of the plate?") is weak, because the only interaction is one dot product.

## Cross-attention into a frozen LM (Flamingo)

$$
y\leftarrow y+\tanh(\alpha_{\text{xattn}})\cdot\mathrm{XAttn}\big(q=y,\ kv=Z\big),\qquad y\leftarrow y+\tanh(\alpha_{\text{ff}})\cdot\mathrm{FFW}(y),\qquad Z=\mathrm{Resampler}(X_v)\in\mathbb R^{64\times d}
$$

- $y$: text hidden states entering a frozen LM layer
- $X_v$: features from a frozen vision encoder (an NFNet-F6 pre-trained contrastively); their number depends on image size and video length
- $\mathrm{Resampler}$: Perceiver resampler: 64 learned latent queries cross-attend to $X_v$ and output a fixed 64 visual tokens
- $\alpha$: learnable scalars initialised to 0, so $\tanh(\alpha)=0$ and the model starts exactly as the frozen LM

Alayrac et al., "Flamingo: a Visual Language Model for Few-Shot Learning" (NeurIPS 2022), kept a pre-trained Chinchilla LM (70B for Flamingo-80B) and the vision encoder frozen, and trained only the resampler and the inserted GATED XATTN-DENSE layers. The tanh gates matter: random cross-attention added to a frozen LM at full strength would destroy its predictions at step 0. Starting from zero, the gates let the visual signal grow as it becomes useful. Inputs are interleaved web documents (M3W, 43M pages), image–text and video–text pairs. A per-image mask lets each text token cross-attend only to the visual tokens of the most recent preceding image. The LM self-attention still sees all previous text, so the model can use in-context examples. Flamingo was evaluated on 16 image and video benchmarks and set a new few-shot state of the art on the 9 that had published few-shot results. On 6 of them it beat fine-tuned models while using only 32 task-specific examples.

## Projection into the LM (LLaVA)

$$
H_v=W\cdot Z_v,\qquad Z_v=g(X_v)\in\mathbb R^{N\times1024},\qquad \text{LM input}=\big[H_v;\ \mathrm{Embed}(\text{prompt})\big]
$$

- $g$: a frozen CLIP ViT-L/14 image encoder; $Z_v$ are its patch ("grid") features
- $W$: a linear map to the LM width in LLaVA; a 2-layer MLP with GELU in LLaVA-1.5
- $N$: $(224/14)^2=256$ in LLaVA, $(336/14)^2=576$ in LLaVA-1.5

Liu, Li, Wu and Lee, "Visual Instruction Tuning" (NeurIPS 2023), showed that the simplest connector works if the training data is right. Stage 1 trains only $W$ on 595K filtered CC3M image–caption pairs, aligning visual features with the word-embedding space. Stage 2 fine-tunes $W$ and the LM (Vicuna) on 158K GPT-4-generated instruction-following conversations about images. The projected patches are ordinary input tokens to a **Decoder-only transformer**, so the causal mask covers them as well: text tokens attend to every image token, and image tokens never attend to text. LLaVA-1.5 ("Improved Baselines with Visual Instruction Tuning", CVPR 2024) used an MLP connector and 336-px input, and became a common baseline. BLIP-2 (Li et al., ICML 2023) sits between Flamingo and LLaVA. Its Q-Former compresses the image into 32 learned query tokens, which are then projected into a frozen LM's input.

> Figure: The attention mask over one toy image and the caption 'a dog naps on a red sofa'. Blue entries are same-modality, orange cross-modal, grey masked. With 6 patches the LLaVA mask is causal over all 13 tokens: 49 same-modality and 42 cross-modal entries, and 78 masked, including every image-to-text entry. Single-stream fills the whole square. ViLBERT splits it into a co-attention sub-layer (only the off-diagonal blocks) and a self-attention sub-layer (only the diagonal blocks). CLIP has no orange at all. In Flamingo mode the patches are replaced by 3 resampler latents that only serve as keys and values, and the gate slider fades them by tanh(α). The right panel counts the same patterns at real scale with 14-px patches and 32 text tokens. At 336² LLaVA puts 576 image tokens before the text, 185,136 causal scores per layer, of which 18,432 are text-to-image. Single-stream needs 369,664. Flamingo's LM sees 64 latents at any resolution: 2,576 scores, 2,048 of them cross-attention. Moving to 672² multiplies LLaVA's image tokens by 4, to 2,304, and its scores to 2,729,616.

## Worked example: CLIP loss for a batch of three

Cosine similarities uᵢᵀvⱼ (rows images, columns captions; diagonal = matching pairs) and the softmax rows at τ = 0.07 (computed in Node)

| image | caption 1 | caption 2 | caption 3 | row softmax (image → text) |
| --- | --- | --- | --- | --- |
| image 1 | **0.30** | 0.10 | 0.05 | 0.9212, 0.0529, 0.0259 |
| image 2 | 0.12 | **0.28** | 0.20 | 0.0716, 0.7039, 0.2245 |
| image 3 | 0.02 | 0.15 | **0.25** | 0.0293, 0.1877, 0.7830 |

**From similarities to the symmetric loss**

1. Divide by $\tau=0.07$: image 2's logits are $(1.714,\ 4.000,\ 2.857)$. Similarities that differ by only 0.08 (0.28 vs 0.20) become a probability of 0.7039 against 0.2245. The small temperature makes the loss sensitive to small cosine gaps.
2. Image→text cross-entropies $-\log$ of the diagonal: $0.0821,\ 0.3511,\ 0.2446$, mean $0.2259$.
3. Text→image uses the columns. For caption 3 the column softmax is $(0.0167,\ 0.1267,\ 0.6464)$, so its loss is $0.4363$. The three column losses are $0.0905,\ 0.2091,\ 0.4363$, mean $0.2453$.
4. $\mathcal L=(0.2259+0.2453)/2=$ **0.2356**. A random encoder would give about $\log3=1.0986$; with CLIP's batch of 32,768 the chance level is $\log32768=10.4$.
5. Token budget for LLaVA-1.5: $336/14=24$, $24^2=$ **576** visual tokens per image. Its MLP connector for the 7B LM ($1024\to4096\to4096$) has $1024\cdot4096+4096+4096\cdot4096+4096=20{,}979{,}712$ parameters, about 0.3% of the LM.

```python
import torch, torch.nn as nn, torch.nn.functional as F

# 1) CLIP: symmetric contrastive loss over B matched pairs
def clip_loss(img_emb, txt_emb, logit_scale):
    u = F.normalize(img_emb, dim=-1)               # (B, D)
    v = F.normalize(txt_emb, dim=-1)               # (B, D)
    logits = logit_scale * u @ v.t()               # (B, B); logit_scale = 1/tau
    target = torch.arange(len(u))
    return (F.cross_entropy(logits, target) + F.cross_entropy(logits.t(), target)) / 2

print(clip_loss(torch.randn(4, 512), torch.randn(4, 512), 1 / 0.07).shape)   # torch.Size([])

# 2) LLaVA-1.5: MLP projector from CLIP ViT-L/14-336 patch features to the LM width
proj = nn.Sequential(nn.Linear(1024, 4096), nn.GELU(), nn.Linear(4096, 4096))
patches = torch.randn(1, 576, 1024)                # (336/14)^2 = 576 patch features
txt = torch.randn(1, 32, 4096)                     # embedded prompt tokens
llm_input = torch.cat([proj(patches), txt], dim=1)
print(llm_input.shape)                             # torch.Size([1, 608, 4096])
print(sum(p.numel() for p in proj.parameters()))   # 20979712

# 3) Flamingo-style gated cross-attention, inserted before a frozen LM layer
class GatedXAttn(nn.Module):
    def __init__(self, d, heads):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.xattn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
        self.a_attn = nn.Parameter(torch.zeros(1))  # tanh(0) = 0: identity at init
        self.a_ff = nn.Parameter(torch.zeros(1))

    def forward(self, y, z):                        # y: (B, T, d) text, z: (B, 64, d) latents
        y = y + self.a_attn.tanh() * self.xattn(self.ln1(y), z, z, need_weights=False)[0]
        return y + self.a_ff.tanh() * self.ff(self.ln2(y))

blk = GatedXAttn(512, 8)
y = torch.randn(2, 32, 512)
out = blk(y, torch.randn(2, 64, 512))
print(out.shape, torch.equal(out, y))              # torch.Size([2, 32, 512]) True
```

Three connectors in PyTorch: CLIP loss, a LLaVA-1.5 projector, a Flamingo gated cross-attention block

> **Visual tokens dominate the sequence:** At $336^2$ one image costs 576 tokens in LLaVA-1.5, more than most prompts. Ten images, or a few seconds of video sampled per frame, fill a 4k context. Attention cost grows with the square of the total length (see **Causal attention**). Raising the resolution from 336 to 672 quadruples the image tokens. That is why Flamingo resamples to 64 tokens, BLIP-2 to 32 queries, and later models pool, merge or tile patches.

> **Freeze, then connect:** Flamingo, BLIP-2 and LLaVA's first stage all keep pre-trained unimodal models frozen and train only a small connector. Each side already holds most of the knowledge: CLIP-style encoders know visual concepts, and the LM knows language and the world. The connector only has to align the two representation spaces. That is cheap and keeps the LM's abilities intact. Zero-initialised gates (Flamingo) and a projection-only first stage (LLaVA) both make the connection start from "no effect".

> **Contrastive and generative models are complementary:** A CLIP-style dual encoder is the usual **image tokenizer** for generative multimodal LMs. LLaVA's vision tower is CLIP ViT-L/14. The contrastive pre-training gives patch features that already line up with language, and the LM adds the fine-grained, token-level interaction the dual encoder lacks. For retrieval at scale the dual encoder stays the right tool, because cross-attention models must rerun for every image–text pair.

**Strengths**

- One architecture for all modalities once each is tokenised; attention learns the cross-modal alignment
- Dual encoders: precomputable embeddings, zero-shot transfer, cheap retrieval
- Cross-attention and projection reuse frozen LLMs and vision encoders, so training is cheap
- Single-stream and LM-based fusion support fine-grained reasoning about image regions and text

**Limitations**

- Visual tokens make sequences long; cost grows quadratically with resolution and number of images
- Dual encoders cannot model fine-grained relations; joint models cannot index the two sides separately
- Performance depends heavily on the size and quality of paired data (web alt-text is noisy)
- Language priors can override the image: models answer from text statistics (hallucination)

**Question:** Which design lets you embed a billion images once and then search them with any new text query at the cost of a dot product per image?

- A. Single-stream encoder (UNITER)
- B. Co-attention (ViLBERT)
- C. Dual encoder (CLIP)
- D. Projection into an LM (LLaVA)

**Answer:** C. Dual encoder (CLIP) — Only a dual encoder computes the image embedding without looking at the text. The other designs mix image and text tokens inside the network, so every new (image, query) pair needs a fresh forward pass. CLIP's score is $u_i^\top v_j/\tau$, a nearest-neighbour search over precomputed $u_i$.

**Question:** Flamingo initialises the gate α of each new cross-attention layer to 0. What is the effect?

- A. The visual tokens are ignored throughout training
- B. At initialisation the model computes exactly what the frozen LM computes; the visual contribution tanh(α)·XAttn grows as α is learned
- C. The cross-attention weights are frozen
- D. The resampler outputs zero vectors

**Answer:** B. At initialisation the model computes exactly what the frozen LM computes; the visual contribution tanh(α)·XAttn grows as α is learned — $\tanh(0)=0$, so $y\leftarrow y+0\cdot\mathrm{XAttn}$ and $y\leftarrow y+0\cdot\mathrm{FFW}$ leave the LM's hidden states unchanged at step 0. The gradient with respect to $\alpha$ is not zero, because $\tanh'(0)=1$, so the gates open as the visual signal becomes useful. The code above checks the identity with `torch.equal`.

**Key points**

- Tokenise each modality: image patches $N=(H/P)^2$ (576 at $336^2$, $P=14$), audio log-Mel frames (Whisper: 30 s → 1500 tokens), then add position and modality information.
- Fusion: single-stream joint attention; co-attention (ViLBERT); dual encoder with CLIP loss $s_{ij}=u_i^\top v_j/\tau$; gated cross-attention $y+\tanh(\alpha)\,\mathrm{XAttn}(y,Z)$ (Flamingo); projection $H_v=WZ_v$ into the LM (LLaVA).
- CLIP: 400M pairs, symmetric InfoNCE, τ initialised 0.07, 76.2% zero-shot ImageNet with ViT-L/14@336.
- Flamingo: frozen vision encoder and LM, Perceiver resampler to 64 tokens, zero-initialised tanh gates.
- LLaVA: CLIP ViT-L/14 patch features → linear (MLP in 1.5) → LM tokens; two-stage training on 595K captions then 158K instructions.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
