# Multimodal representation learning

> Multimodal representation learning maps modalities into joint or coordinated vector spaces. Contrastive InfoNCE alignment dominates but leaves a modality gap; fusion can be early, mid (cross-attention) or late.

Source: https://www.mlfoundry.in/learn/multimodal-representation-learning  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: advanced · 8 min read · updated 2026-10-15

**Multimodal representation learning:** learns vector representations from two or more modalities (images, text, audio, depth, sensor streams) so that information shared between them is captured, and so that one modality can be used to retrieve, label or predict another. The standard taxonomy (Baltrušaitis, Ahuja and Morency, "Multimodal Machine Learning: A Survey and Taxonomy", TPAMI 2019) separates **joint** representations, one vector computed from all modalities, from **coordinated** representations, one vector per modality, tied by a similarity constraint. An early deep example is the bimodal audio–video autoencoder of Ngiam et al. (ICML 2011).

The modalities are heterogeneous: a 224×224 image, a 77-token caption and a 10-second waveform have nothing in common at the input. Pairing is weak, because a web caption describes only part of its image. Some modalities are often missing at test time. A good representation must keep what the modalities share, keep what each adds, and degrade gracefully when one is absent.

## Joint versus coordinated spaces

> Figure: (a) A joint representation runs one network over both inputs and produces a single vector z for the pair. (b) Coordinated representations encode each modality separately and train u and v to agree, usually with InfoNCE. (c) Fusion depth: early fusion mixes the inputs, mid fusion lets tokens of each modality attend to the other, late fusion only averages predictions. (d) ImageBind trains each of text, audio, depth, thermal and IMU encoders against images only; text–audio alignment then emerges without any text–audio pairs.

Joint representations suit tasks that need fine interactions between modalities, such as VQA, where the question decides which image region matters. Coordinated representations suit retrieval: you can encode a billion images once and search them with a text vector. DeViSE (Frome et al., 2013) mapped image features onto word2vec label embeddings. **CLIP** made the coordinated approach dominant by training both encoders contrastively on 400M pairs.

## Contrastive alignment

With a batch of $N$ pairs and L2-normalised embeddings, the symmetric contrastive loss treats each true pair as the positive and the other $N-1$ items in the batch as negatives (see **InfoNCE loss** and **Contrastive language learning**):

$$
\mathcal L=-\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{e^{u_i^\top v_i/\tau}}{\sum_{j}e^{u_i^\top v_j/\tau}}+\log\frac{e^{u_i^\top v_i/\tau}}{\sum_{j}e^{u_j^\top v_i/\tau}}\right],\qquad I(U;V)\ \ge\ \log N-\mathcal L_{\text{one direction}}
$$

- $u_i,\ v_i$: unit-norm embeddings of the two modalities of pair $i$
- $\tau$: temperature; CLIP learns it, starting from 0.07, with the logit scale 1/τ capped at 100
- $\log N$: the loss bounds the mutual information from below (van den Oord et al., 2018), so larger batches can certify more shared information

Wang and Isola (ICML 2020) split what such a loss optimises into two parts. **Alignment** pulls positive pairs together. **Uniformity** spreads all embeddings over the hypersphere so that they keep information:

$$
\mathcal L_{\text{align}}=\mathbb E_{(x,y)\sim p_{\text{pos}}}\big\|f(x)-g(y)\big\|_2^2,\qquad \mathcal L_{\text{uniform}}=\log\ \mathbb E_{x,x'\sim p_{\text{data}}}\,e^{-2\|f(x)-f(x')\|_2^2}
$$

- $f,\ g$: the two modality encoders, with normalised outputs
- $\mathcal L_{\text{uniform}}$: log of the average Gaussian potential between pairs; low when points are spread out

## The modality gap

Liang, Zhang, Kwon, Yeung and Zou ("Mind the Gap", NeurIPS 2022) found that in CLIP and other contrastive models, image embeddings and text embeddings occupy two separate narrow cones. The gap is the distance between the modality means, $\Delta=\|\bar u-\bar v\|$. They traced it to two causes. Randomly initialised deep encoders already map all inputs into a narrow cone, and each encoder has its own cone. Contrastive training at low temperature then preserves the separation. Shifting embeddings along $\Delta$ changed zero-shot accuracy and fairness, so the gap is not harmless.

A simple model shows why the gap goes unnoticed. Suppose image $i$ is $(s_i,\gamma)/\sqrt{1+\gamma^2}$ and its caption is $(s_i',-\gamma)/\sqrt{1+\gamma^2}$, where the offset $\gamma$ is orthogonal to the semantic part $s$. Then $u_i^\top v_j=(s_i^\top s_j'-\gamma^2)/(1+\gamma^2)$. This is increasing in $s_i^\top s_j'$, so cross-modal rankings do not change. The offset only rescales the logits, which acts like a temperature of $\tau(1+\gamma^2)$.

Worked example with γ = 0.75 (1 + γ² = 1.5625), concepts 45° apart on a circle (computed in Python)

| Pair | Semantic cos sᵀs′ | Embedding cosine | Formula |
| --- | --- | --- | --- |
| image i, its caption | 1 | 0.280 | (1 − 0.5625) / 1.5625 |
| image i, neighbour's caption | 0.7071 | 0.093 | (0.7071 − 0.5625) / 1.5625 |
| image i, neighbour's image | 0.7071 | 0.813 | (0.7071 + 0.5625) / 1.5625 |
| image i, opposite caption | −1 | −1.000 | (−1 − 0.5625) / 1.5625 |
| gap ‖ū − v̄‖ |  | 1.200 | 2γ / √(1 + γ²) |

The matched caption is still the nearest caption (0.280 against 0.093), so image→text retrieval is perfect. But a neighbouring image is much closer (0.813). In a mixed index of images and captions, an image query returns images first. Thresholds on absolute cosine values also become modality-specific. Subtracting each modality's mean and re-normalising removes the offset in this model.

> Figure: Eight concepts on a unit sphere, with images pushed up and texts pushed down by the offset γ, plus angular matching noise σ. Defaults (γ = 0.75, σ = 0.25, τ = 0.1): gap 1.201, matched cosine 0.238, image–image cosine 0.274. Cross-modal R@1 is 7/8 in both directions, but R@1 in a mixed image+text pool is 0/8, and the InfoNCE loss is 0.556 (chance is log 8 = 2.079). Centring each modality gives gap 0.038, matched cosine 0.936, mixed R@1 7/8 and loss 0.492. At τ = 0.02 the gap is favoured: the raw embeddings score 0.685 and γ = 1.5 scores 0.483, while centring raises the loss to 1.115.

## Binding many modalities, and other objectives

ImageBind (Girdhar et al., CVPR 2023) aligns six modalities (images and video, text, audio, depth, thermal, IMU) in one space. It uses only pairs that contain an image: image–text from the web, and video–audio, image–depth, image–thermal and video–IMU pairs from naturally co-occurring data. Each new encoder is trained with InfoNCE against image embeddings. Because images are the hub, audio and text become aligned although they were never paired. This allows zero-shot audio classification with text prompts and audio-to-image retrieval.

Contrastive learning is not the only option. FLAVA (Singh et al., CVPR 2022) combines contrastive loss with masked image, masked language and **masked multimodal modelling** on a fusion encoder. BEiT-3 (Wang et al., CVPR 2023) treats images as a foreign language and pre-trains a shared Multiway transformer with masked data modelling alone. Masked objectives learn fine-grained joint features. Contrastive objectives learn retrieval-ready coordinated features. Many models use both.

**How representations are evaluated**

- **Linear probe:** freeze the encoder, fit logistic regression on its features for a labelled dataset; measures how linearly separable the information is
- **Zero-shot:** classify by nearest text embedding of prompts such as “a photo of a {label}”; possible only with coordinated spaces
- **Retrieval:** Recall@K (K = 1, 5, 10) for image→text and text→image, on Flickr30K (1K test images) and MS-COCO (5K test images, 5 captions each)
- **Fine-tuning:** train the whole model on a downstream task (VQA, captioning) to test the representation as an initialisation

```python
import torch, torch.nn.functional as F
from transformers import CLIPModel, CLIPProcessor

name = "openai/clip-vit-base-patch32"
model, proc = CLIPModel.from_pretrained(name).eval(), CLIPProcessor.from_pretrained(name)

with torch.no_grad():
    batch = proc(text=captions, images=images, return_tensors="pt", padding=True)
    out = model(**batch)
u, v = out.image_embeds, out.text_embeds          # already L2-normalised, shape (N, 512)

gap = (u.mean(0) - v.mean(0)).norm()
def r_at_1(a, b):                                  # a[i] should retrieve b[i]
    return ((a @ b.T).argmax(1) == torch.arange(len(a))).float().mean()

uc, vc = F.normalize(u - u.mean(0), dim=-1), F.normalize(v - v.mean(0), dim=-1)
print(f"gap {gap:.3f}  R@1 {r_at_1(u, v):.3f}  centred R@1 {r_at_1(uc, vc):.3f}")
print("matched cos", (u * v).sum(1).mean().item(), "image-image cos", (u @ u.T).mean().item())
```

Measuring the modality gap and the effect of centring on CLIP embeddings with Hugging Face Transformers (not run here; supply your own paired images and captions)

> **Do not compare cosines across modalities:** A text–image cosine of 0.3 may be a strong match in CLIP while an image–image cosine of 0.8 is unremarkable, because of the gap. Mixed-modality indexes, fixed similarity thresholds and arithmetic such as "image + text vector" all need per-modality centring or calibration first.

> **Images are the natural hub:** Paired data for most modality combinations is scarce, but almost everything co-occurs with images or video. Aligning each modality to a strong image space, as ImageBind does, buys alignment between all pairs. The quality of the hub space then bounds everything bound to it.

**Question:** Every text embedding in a coordinated space is shifted by the same vector orthogonal to all semantic directions, then re-normalised. What happens to text→image Recall@1?

- A. Nothing: every image's similarity to a text is scaled and offset in the same way, so the ranking is unchanged
- B. It drops to chance, because the matched cosines fall
- C. It improves, because the texts become more uniform
- D. It cannot be computed without retraining

**Answer:** A. Nothing: every image's similarity to a text is scaled and offset in the same way, so the ranking is unchanged — The images are untouched and the shift is orthogonal to them, so for a fixed text every image's cosine becomes $s^\top s'/\sqrt{1+\gamma^2}$: the same increasing function of semantic similarity. Rankings, and so Recall@1, are preserved. What changes is the absolute value of the cosines and the comparison with same-modality neighbours.

**Question:** Which fusion choice keeps working when the audio stream is missing at test time, without retraining?

- A. Early fusion of raw inputs
- B. Late fusion of per-modality predictions
- C. A joint encoder with cross-attention in every layer
- D. Concatenating audio and video features before the classifier

**Answer:** B. Late fusion of per-modality predictions — Late fusion trains a separate predictor per modality and only averages their outputs, so dropping one modality just removes one term. The other options feed the missing input into shared layers that were trained to expect it.

**Key points**

- Joint representations encode modalities together (fine interactions, need all inputs). Coordinated representations encode each separately and align them (retrieval, zero-shot).
- Symmetric InfoNCE with temperature $\tau$ is the standard alignment loss; it lower-bounds mutual information by $\log N-\mathcal L$. Wang and Isola split it into alignment and uniformity.
- The modality gap (Liang et al., 2022): image and text embeddings sit in separate cones. An offset orthogonal to semantics keeps cross-modal rankings but breaks mixed-modality search and absolute thresholds.
- ImageBind binds six modalities through images alone; text–audio alignment emerges.
- Fusion depth: early (raw interactions), mid (cross-attention, bottlenecks), late (robust to missing inputs). Evaluate with linear probes, zero-shot and Recall@K.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
