Definition
Multimodal representation learning learns vector representations from two or more modalities (images, text, audio, depth, sensor streams) so that information shared between them is captured, and so that one modality can be used to retrieve, label or predict another. The standard taxonomy (Baltrušaitis, Ahuja and Morency, "Multimodal Machine Learning: A Survey and Taxonomy", TPAMI 2019) separates joint representations, one vector computed from all modalities, from coordinated representations, one vector per modality, tied by a similarity constraint. An early deep example is the bimodal audio–video autoencoder of Ngiam et al. (ICML 2011).
The modalities are heterogeneous: a 224×224 image, a 77-token caption and a 10-second waveform have nothing in common at the input. Pairing is weak, because a web caption describes only part of its image. Some modalities are often missing at test time. A good representation must keep what the modalities share, keep what each adds, and degrade gracefully when one is absent.
Joint versus coordinated spaces
Joint representations suit tasks that need fine interactions between modalities, such as VQA, where the question decides which image region matters. Coordinated representations suit retrieval: you can encode a billion images once and search them with a text vector. DeViSE (Frome et al., 2013) mapped image features onto word2vec label embeddings. CLIP made the coordinated approach dominant by training both encoders contrastively on 400M pairs.
Contrastive alignment
With a batch of pairs and L2-normalised embeddings, the symmetric contrastive loss treats each true pair as the positive and the other items in the batch as negatives (see InfoNCE loss and Contrastive language learning):
- unit-norm embeddings of the two modalities of pair
- temperature; CLIP learns it, starting from 0.07, with the logit scale 1/τ capped at 100
- the loss bounds the mutual information from below (van den Oord et al., 2018), so larger batches can certify more shared information
Wang and Isola (ICML 2020) split what such a loss optimises into two parts. Alignment pulls positive pairs together. Uniformity spreads all embeddings over the hypersphere so that they keep information:
- the two modality encoders, with normalised outputs
- log of the average Gaussian potential between pairs; low when points are spread out
The modality gap
Liang, Zhang, Kwon, Yeung and Zou ("Mind the Gap", NeurIPS 2022) found that in CLIP and other contrastive models, image embeddings and text embeddings occupy two separate narrow cones. The gap is the distance between the modality means, . They traced it to two causes. Randomly initialised deep encoders already map all inputs into a narrow cone, and each encoder has its own cone. Contrastive training at low temperature then preserves the separation. Shifting embeddings along changed zero-shot accuracy and fairness, so the gap is not harmless.
A simple model shows why the gap goes unnoticed. Suppose image is and its caption is , where the offset is orthogonal to the semantic part . Then . This is increasing in , so cross-modal rankings do not change. The offset only rescales the logits, which acts like a temperature of .
| Pair | Semantic cos sᵀs′ | Embedding cosine | Formula |
|---|---|---|---|
| image i, its caption | 1 | 0.280 | (1 − 0.5625) / 1.5625 |
| image i, neighbour's caption | 0.7071 | 0.093 | (0.7071 − 0.5625) / 1.5625 |
| image i, neighbour's image | 0.7071 | 0.813 | (0.7071 + 0.5625) / 1.5625 |
| image i, opposite caption | −1 | −1.000 | (−1 − 0.5625) / 1.5625 |
| gap ‖ū − v̄‖ | 1.200 | 2γ / √(1 + γ²) |
The matched caption is still the nearest caption (0.280 against 0.093), so image→text retrieval is perfect. But a neighbouring image is much closer (0.813). In a mixed index of images and captions, an image query returns images first. Thresholds on absolute cosine values also become modality-specific. Subtracting each modality's mean and re-normalising removes the offset in this model.
Binding many modalities, and other objectives
ImageBind (Girdhar et al., CVPR 2023) aligns six modalities (images and video, text, audio, depth, thermal, IMU) in one space. It uses only pairs that contain an image: image–text from the web, and video–audio, image–depth, image–thermal and video–IMU pairs from naturally co-occurring data. Each new encoder is trained with InfoNCE against image embeddings. Because images are the hub, audio and text become aligned although they were never paired. This allows zero-shot audio classification with text prompts and audio-to-image retrieval.
Contrastive learning is not the only option. FLAVA (Singh et al., CVPR 2022) combines contrastive loss with masked image, masked language and masked multimodal modelling on a fusion encoder. BEiT-3 (Wang et al., CVPR 2023) treats images as a foreign language and pre-trains a shared Multiway transformer with masked data modelling alone. Masked objectives learn fine-grained joint features. Contrastive objectives learn retrieval-ready coordinated features. Many models use both.
How representations are evaluated
- Linear probe
- freeze the encoder, fit logistic regression on its features for a labelled dataset; measures how linearly separable the information is
- Zero-shot
- classify by nearest text embedding of prompts such as “a photo of a {label}”; possible only with coordinated spaces
- Retrieval
- Recall@K (K = 1, 5, 10) for image→text and text→image, on Flickr30K (1K test images) and MS-COCO (5K test images, 5 captions each)
- Fine-tuning
- train the whole model on a downstream task (VQA, captioning) to test the representation as an initialisation
1import torch, torch.nn.functional as F2from transformers import CLIPModel, CLIPProcessor34name = "openai/clip-vit-base-patch32"5model, proc = CLIPModel.from_pretrained(name).eval(), CLIPProcessor.from_pretrained(name)67with torch.no_grad():8 batch = proc(text=captions, images=images, return_tensors="pt", padding=True)9 out = model(**batch)10u, v = out.image_embeds, out.text_embeds # already L2-normalised, shape (N, 512)1112gap = (u.mean(0) - v.mean(0)).norm()13def r_at_1(a, b): # a[i] should retrieve b[i]14 return ((a @ b.T).argmax(1) == torch.arange(len(a))).float().mean()1516uc, vc = F.normalize(u - u.mean(0), dim=-1), F.normalize(v - v.mean(0), dim=-1)17print(f"gap {gap:.3f} R@1 {r_at_1(u, v):.3f} centred R@1 {r_at_1(uc, vc):.3f}")18print("matched cos", (u * v).sum(1).mean().item(), "image-image cos", (u @ u.T).mean().item())Check yourself
Every text embedding in a coordinated space is shifted by the same vector orthogonal to all semantic directions, then re-normalised. What happens to text→image Recall@1?
Check yourself
Which fusion choice keeps working when the audio stream is missing at test time, without retraining?
Remember
- ✓Joint representations encode modalities together (fine interactions, need all inputs). Coordinated representations encode each separately and align them (retrieval, zero-shot).
- ✓Symmetric InfoNCE with temperature is the standard alignment loss; it lower-bounds mutual information by . Wang and Isola split it into alignment and uniformity.
- ✓The modality gap (Liang et al., 2022): image and text embeddings sit in separate cones. An offset orthogonal to semantics keeps cross-modal rankings but breaks mixed-modality search and absolute thresholds.
- ✓ImageBind binds six modalities through images alone; text–audio alignment emerges.
- ✓Fusion depth: early (raw interactions), mid (cross-attention, bottlenecks), late (robust to missing inputs). Evaluate with linear probes, zero-shot and Recall@K.