Multimodal representation learning

Multimodal representation learning maps modalities into joint or coordinated vector spaces. Contrastive InfoNCE alignment dominates but leaves a modality gap; fusion can be early, mid (cross-attention) or late.

8 min read832 wordsadvanced0 viewsupdated 2026-10-15

Definition

Multimodal representation learning learns vector representations from two or more modalities (images, text, audio, depth, sensor streams) so that information shared between them is captured, and so that one modality can be used to retrieve, label or predict another. The standard taxonomy (Baltrušaitis, Ahuja and Morency, "Multimodal Machine Learning: A Survey and Taxonomy", TPAMI 2019) separates joint representations, one vector computed from all modalities, from coordinated representations, one vector per modality, tied by a similarity constraint. An early deep example is the bimodal audio–video autoencoder of Ngiam et al. (ICML 2011).

The modalities are heterogeneous: a 224×224 image, a 77-token caption and a 10-second waveform have nothing in common at the input. Pairing is weak, because a web caption describes only part of its image. Some modalities are often missing at test time. A good representation must keep what the modalities share, keep what each adds, and degrade gracefully when one is absent.

Joint versus coordinated spaces

(a) joint representationimage x_Itext x_Tf(x_I, x_T)zone vector for the pairneeds both inputs at test timee.g. fusion encoders for VQA(b) coordinated representationsimage x_Itext x_Tf_If_Tuvsim(u, v)each modality encoded alone; InfoNCE pullspairs together, pushes the batch aparte.g. DeViSE, CLIP, ImageBind(c) where to fuseearlymid (cross-attn)lateITone encoderŷITencencŷITencencŷ_Iŷ_Tavg → ŷearly: low-level interactions, needs aligned inputsmid: tokens attend across modalities (ViLBERT,Flamingo, attention bottlenecks)late: robust to a missing modality, no interaction(d) ImageBind: bind every modality to imagesemergentimage/videotextaudiodepththermalIMUgreen = trainedInfoNCE on (image, X)
fig.(a) A joint representation runs one network over both inputs and produces a single vector z for the pair. (b) Coordinated representations encode each modality separately and train u and v to agree, usually with InfoNCE. (c) Fusion depth: early fusion mixes the inputs, mid fusion lets tokens of each modality attend to the other, late fusion only averages predictions. (d) ImageBind trains each of text, audio, depth, thermal and IMU encoders against images only; text–audio alignment then emerges without any text–audio pairs.

Joint representations suit tasks that need fine interactions between modalities, such as VQA, where the question decides which image region matters. Coordinated representations suit retrieval: you can encode a billion images once and search them with a text vector. DeViSE (Frome et al., 2013) mapped image features onto word2vec label embeddings. CLIP made the coordinated approach dominant by training both encoders contrastively on 400M pairs.

Contrastive alignment

With a batch of NN pairs and L2-normalised embeddings, the symmetric contrastive loss treats each true pair as the positive and the other N−1N-1 items in the batch as negatives (see InfoNCE loss and Contrastive language learning):

L=−12N∑i=1N[log⁡eui⊤vi/τ∑jeui⊤vj/τ+log⁡eui⊤vi/τ∑jeuj⊤vi/τ],I(U;V) ≥ log⁡N−Lone direction\mathcal L=-\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{e^{u_i^\top v_i/\tau}}{\sum_{j}e^{u_i^\top v_j/\tau}}+\log\frac{e^{u_i^\top v_i/\tau}}{\sum_{j}e^{u_j^\top v_i/\tau}}\right],\qquad I(U;V)\ \ge\ \log N-\mathcal L_{\text{one direction}}
ui, viu_i,\ v_i
unit-norm embeddings of the two modalities of pair ii
τ\tau
temperature; CLIP learns it, starting from 0.07, with the logit scale 1/τ capped at 100
log⁡N\log N
the loss bounds the mutual information from below (van den Oord et al., 2018), so larger batches can certify more shared information

Wang and Isola (ICML 2020) split what such a loss optimises into two parts. Alignment pulls positive pairs together. Uniformity spreads all embeddings over the hypersphere so that they keep information:

Lalign=E(x,y)∼ppos∥f(x)−g(y)∥22,Luniform=log⁡ Ex,x′∼pdata e−2∥f(x)−f(x′)∥22\mathcal L_{\text{align}}=\mathbb E_{(x,y)\sim p_{\text{pos}}}\big\|f(x)-g(y)\big\|_2^2,\qquad \mathcal L_{\text{uniform}}=\log\ \mathbb E_{x,x'\sim p_{\text{data}}}\,e^{-2\|f(x)-f(x')\|_2^2}
f, gf,\ g
the two modality encoders, with normalised outputs
Luniform\mathcal L_{\text{uniform}}
log of the average Gaussian potential between pairs; low when points are spread out

The modality gap

Liang, Zhang, Kwon, Yeung and Zou ("Mind the Gap", NeurIPS 2022) found that in CLIP and other contrastive models, image embeddings and text embeddings occupy two separate narrow cones. The gap is the distance between the modality means, Δ=∥uˉ−vˉ∥\Delta=\|\bar u-\bar v\|. They traced it to two causes. Randomly initialised deep encoders already map all inputs into a narrow cone, and each encoder has its own cone. Contrastive training at low temperature then preserves the separation. Shifting embeddings along Δ\Delta changed zero-shot accuracy and fairness, so the gap is not harmless.

A simple model shows why the gap goes unnoticed. Suppose image ii is (si,γ)/1+γ2(s_i,\gamma)/\sqrt{1+\gamma^2} and its caption is (si′,−γ)/1+γ2(s_i',-\gamma)/\sqrt{1+\gamma^2}, where the offset γ\gamma is orthogonal to the semantic part ss. Then ui⊤vj=(si⊤sj′−γ2)/(1+γ2)u_i^\top v_j=(s_i^\top s_j'-\gamma^2)/(1+\gamma^2). This is increasing in si⊤sj′s_i^\top s_j', so cross-modal rankings do not change. The offset only rescales the logits, which acts like a temperature of τ(1+γ2)\tau(1+\gamma^2).

PairSemantic cos sᵀs′Embedding cosineFormula
image i, its caption10.280(1 − 0.5625) / 1.5625
image i, neighbour's caption0.70710.093(0.7071 − 0.5625) / 1.5625
image i, neighbour's image0.70710.813(0.7071 + 0.5625) / 1.5625
image i, opposite caption−1−1.000(−1 − 0.5625) / 1.5625
gap ‖ū − v̄‖1.2002γ / √(1 + γ²)
Worked example with γ = 0.75 (1 + γ² = 1.5625), concepts 45° apart on a circle (computed in Python)

The matched caption is still the nearest caption (0.280 against 0.093), so image→text retrieval is perfect. But a neighbouring image is much closer (0.813). In a mixed index of images and captions, an image query returns images first. Thresholds on absolute cosine values also become modality-specific. Subtracting each modality's mean and re-normalising removes the offset in this model.

● images · ■ texts · dashed = true pair1234567812345678images +0.75 and texts −0.75 along z, then normalisedangle noise σ = 0.25 rad · dashed ellipse = equator (z = 0)cos(imageᵢ, textⱼ)rows: imagescols: texts■ +1■ −1modality gap ‖μ_img − μ_txt‖1.201mean cos, matched image–text0.238mean cos, other image–text-0.444mean cos, image–image0.274R@1 image→text / text→image7/8 · 7/8R@1 in a mixed pool (img+txt)0/8InfoNCE at τ = 0.1 (chance 2.079)0.556
post-processing
fig.Eight concepts on a unit sphere, with images pushed up and texts pushed down by the offset γ, plus angular matching noise σ. Defaults (γ = 0.75, σ = 0.25, τ = 0.1): gap 1.201, matched cosine 0.238, image–image cosine 0.274. Cross-modal R@1 is 7/8 in both directions, but R@1 in a mixed image+text pool is 0/8, and the InfoNCE loss is 0.556 (chance is log 8 = 2.079). Centring each modality gives gap 0.038, matched cosine 0.936, mixed R@1 7/8 and loss 0.492. At τ = 0.02 the gap is favoured: the raw embeddings score 0.685 and γ = 1.5 scores 0.483, while centring raises the loss to 1.115.

Binding many modalities, and other objectives

ImageBind (Girdhar et al., CVPR 2023) aligns six modalities (images and video, text, audio, depth, thermal, IMU) in one space. It uses only pairs that contain an image: image–text from the web, and video–audio, image–depth, image–thermal and video–IMU pairs from naturally co-occurring data. Each new encoder is trained with InfoNCE against image embeddings. Because images are the hub, audio and text become aligned although they were never paired. This allows zero-shot audio classification with text prompts and audio-to-image retrieval.

Contrastive learning is not the only option. FLAVA (Singh et al., CVPR 2022) combines contrastive loss with masked image, masked language and masked multimodal modelling on a fusion encoder. BEiT-3 (Wang et al., CVPR 2023) treats images as a foreign language and pre-trains a shared Multiway transformer with masked data modelling alone. Masked objectives learn fine-grained joint features. Contrastive objectives learn retrieval-ready coordinated features. Many models use both.

How representations are evaluated

Linear probe
freeze the encoder, fit logistic regression on its features for a labelled dataset; measures how linearly separable the information is
Zero-shot
classify by nearest text embedding of prompts such as “a photo of a {label}”; possible only with coordinated spaces
Retrieval
Recall@K (K = 1, 5, 10) for image→text and text→image, on Flickr30K (1K test images) and MS-COCO (5K test images, 5 captions each)
Fine-tuning
train the whole model on a downstream task (VQA, captioning) to test the representation as an initialisation
python
1import torch, torch.nn.functional as F
2from transformers import CLIPModel, CLIPProcessor
3
4name = "openai/clip-vit-base-patch32"
5model, proc = CLIPModel.from_pretrained(name).eval(), CLIPProcessor.from_pretrained(name)
6
7with torch.no_grad():
8 batch = proc(text=captions, images=images, return_tensors="pt", padding=True)
9 out = model(**batch)
10u, v = out.image_embeds, out.text_embeds # already L2-normalised, shape (N, 512)
11
12gap = (u.mean(0) - v.mean(0)).norm()
13def r_at_1(a, b): # a[i] should retrieve b[i]
14 return ((a @ b.T).argmax(1) == torch.arange(len(a))).float().mean()
15
16uc, vc = F.normalize(u - u.mean(0), dim=-1), F.normalize(v - v.mean(0), dim=-1)
17print(f"gap {gap:.3f} R@1 {r_at_1(u, v):.3f} centred R@1 {r_at_1(uc, vc):.3f}")
18print("matched cos", (u * v).sum(1).mean().item(), "image-image cos", (u @ u.T).mean().item())
Measuring the modality gap and the effect of centring on CLIP embeddings with Hugging Face Transformers (not run here; supply your own paired images and captions)

Check yourself

Every text embedding in a coordinated space is shifted by the same vector orthogonal to all semantic directions, then re-normalised. What happens to text→image Recall@1?

Check yourself

Which fusion choice keeps working when the audio stream is missing at test time, without retraining?

Remember

  • ✓Joint representations encode modalities together (fine interactions, need all inputs). Coordinated representations encode each separately and align them (retrieval, zero-shot).
  • ✓Symmetric InfoNCE with temperature τ\tau is the standard alignment loss; it lower-bounds mutual information by log⁡N−L\log N-\mathcal L. Wang and Isola split it into alignment and uniformity.
  • ✓The modality gap (Liang et al., 2022): image and text embeddings sit in separate cones. An offset orthogonal to semantics keeps cross-modal rankings but breaks mixed-modality search and absolute thresholds.
  • ✓ImageBind binds six modalities through images alone; text–audio alignment emerges.
  • ✓Fusion depth: early (raw interactions), mid (cross-attention, bottlenecks), late (robust to missing inputs). Evaluate with linear probes, zero-shot and Recall@K.
← back to Deep Learning roadmap