Vision-language models

Vision-language models join an image encoder to language: dual encoders (CLIP) align embeddings, Flamingo cross-attends into a frozen LM, and LLaVA projects 576 CLIP patch tokens into an LLM tuned on visual instructions.

8 min read729 wordscore0 viewsupdated 2026-10-15

Definition

Vision-language model (VLM) a model that takes images and text as input and either scores how well they match or generates text about the image. Three designs dominate. Dual encoders such as CLIP (Radford et al., ICML 2021) embed each modality separately. Cross-attention into a frozen language model, as in Flamingo (Alayrac et al., NeurIPS 2022). Projected visual tokens fed to a decoder-only LLM, as in LLaVA ("Visual Instruction Tuning", Liu, Li, Wu and Lee, NeurIPS 2023).

Tasks: image–text retrieval, zero-shot classification, captioning, visual question answering (VQA), and open-ended chat about images, charts and documents. The hard part is grounding: the text must be about what is actually in the pixels, including counts, positions and small text, and not about what is statistically likely.

Before 2021, captioning used a CNN encoder with an LSTM decoder (Show and Tell, Vinyals et al., 2015). VQA models attended over Faster R-CNN region features ("bottom-up and top-down attention", Anderson et al., CVPR 2018). Fusion encoders such as ViLBERT, LXMERT and UNITER ran one transformer over region features and words, with cross-modal attention in every layer. They were strong on VQA but could not retrieve at scale or write free text.

Three ways to connect vision to language

(a) dual encoder · CLIP: two towers, one shared embedding spaceimagetextimage encoder (ViT)text encodervᵢtⱼsᵢⱼ = cos(vᵢ, tⱼ) / τcontrastive: pair i ↔ i (diagonal)→ retrieval, zero-shot labelsscores pairs; cannot write text(b) cross-attention into a frozen LM · Flamingoimagevision encoderfrozenPerceiver Resampler→ 64 visual tokens zx ← x + tanh(α)·XAttn(x, z)α = 0 at init: starts as the LMtext+ <image>gated xattntrainedLM blockfrozengated xattntrainedLM blockfrozen× Ltrained: resampler + xattn only · data: interleaved image–text web pages · few-shot prompting(c) projected visual tokens into an LLM · LLaVA-1.5image336 × 336CLIP ViT-L/14576 × 10242-layer MLP576 × 4096…576 image | textVicuna LLM→ answer tokens24 × 24 = 576 patches (336 / 14 = 24) · stage 1: train the MLP on 558K image–caption pairsstage 2: MLP + LLM on 665K instruction samples · loss: next-token cross-entropy on answers onlyfusion encoders (ViLBERT, LXMERT, UNITER, ViLT): one joint transformerover region or patch features and words, with cross-modal attention in every layer;strong for VQA and image–text matching, but each pair needs a full forward pass, and no free-formgeneration without an added decoder
fig.(a) A dual encoder scores image–caption pairs through a cosine-similarity matrix. It is cheap for retrieval but cannot generate text. (b) Flamingo keeps both the vision encoder and the language model frozen. A Perceiver Resampler turns the image features into 64 tokens, and new tanh-gated cross-attention layers, inserted between the LM blocks, read them. (c) LLaVA-1.5 projects the 576 patch features of CLIP ViT-L/14 at 336 px (width 1024) with a 2-layer MLP to the LLM width of 4096. It then simply prepends them to the text tokens.

Dual encoders are covered in CLIP and SigLIP. They supply the vision encoder for most generative VLMs too, because contrastive pre-training on hundreds of millions of pairs gives patch features that already line up with language. For the general architecture patterns see Multimodal transformers and Cross-attention.

Flamingo adds trainable layers to a frozen Chinchilla LM (70B parameters in the largest, 80B-parameter Flamingo). Each new layer is gated so that the model starts as the unchanged language model:

x←x+tanh⁡(α)⋅XAttn(x, z),α=0 at initialisationx\leftarrow x+\tanh(\alpha)\cdot\mathrm{XAttn}(x,\,z),\qquad \alpha=0\ \text{at initialisation}
xx
hidden states of the text tokens inside the frozen LM
zz
64 visual tokens from the Perceiver Resampler (a set of learned latents that cross-attend to the image features)
α\alpha
learned scalar per layer; tanh(0) = 0, so training begins from the LM's original behaviour

Trained on web pages with interleaved images and text, Flamingo does few-shot learning in context: a prompt with a few (image, answer) examples followed by a new image. BLIP and BLIP-2 (Li, Li, Savarese and Hoi, ICML 2023) take a similar route. BLIP-2's Q-Former compresses the image into 32 learned query tokens for a frozen LLM.

LLaVA: visual tokens plus instruction tuning

LLaVA uses the simplest connector. Patch features ZvZ_v from a frozen CLIP ViT-L/14 are mapped by a projection WW into the LLM's word-embedding space and placed before the instruction. Training uses the ordinary language-modelling loss, on the answer tokens only:

Hv=W⋅Zv,Zv=g(Xv),L=−∑t∈answerlog⁡pθ(xt∣Hv, Xinstruct, x<t)H_v=W\cdot Z_v,\quad Z_v=g(X_v),\qquad \mathcal L=-\sum_{t\in\text{answer}}\log p_\theta\big(x_t\mid H_v,\,X_{\text{instruct}},\,x_{<t}\big)
gg
CLIP vision encoder; LLaVA takes the grid features of the second-to-last layer, not the [CLS] embedding
WW
one linear layer in LLaVA; a 2-layer MLP with GELU in LLaVA-1.5
HvH_v
N=(R/p)2N=(R/p)^2 visual tokens: 256 at 224 px, 576 at 336 px with patch size 14

The data was the novelty. GPT-4, given only COCO captions and bounding boxes as text, wrote 158K instruction-following samples: conversations, detailed descriptions and multi-step reasoning about each image. Stage 1 trains only the projector on image–caption pairs, to align features. Stage 2 trains the projector and the LLM on the instructions. This recipe, visual instruction tuning, became the template for most open VLMs.

Reference configurations

CLIP ViT-L/14@336px
400M image–text pairs; 76.2% zero-shot ImageNet top-1; 24 × 24 = 576 patch tokens of width 1024
Flamingo-80B
frozen NFNet-F6 vision encoder + frozen Chinchilla 70B LM; 64 visual tokens per image from the Perceiver Resampler
BLIP-2
frozen ViT-g image encoder + Q-Former with 32 queries + frozen OPT or Flan-T5 LLM
LLaVA-1.5
CLIP ViT-L/14 at 336 px + 2-layer MLP + Vicuna-7B/13B; 558K caption pairs (stage 1) and a 665K instruction mix (stage 2); about one day of training on one 8×A100 node; VQAv2 78.5 (7B) and 80.0 (13B)
336×336 image · patch 14 · MLP projector · LLM 7B (32 × 4096)24 × 24 patches of 14 pxvision encoderViT, patch 14→ 576 tokensconnectorMLP projector→ 576 tokenslanguage model7B (32 × 4096)+ 64 text tokensevery patch becomes one LLM token (LLaVA-1.5)token count grows with resolution²LLM input sequence (scale: 4,096 tokens)4096-token contextpatch tokens N = (R/p)²24² = 576visual tokens into the LLM576sequence = visual + text640share of sequence that is image90.0%KV cache (fp16, one sample)320.0 MiBprefill compute8.79 TFLOPvs the text alone10.2×FLOPs ≈ 2·params·L + 4·layers·L²·d · KV bytes = 2·layers·d·2·L · vision encoder cost not included
connector
patch / LLM
fig.The cost of visual tokens in a LLaVA-style model. At the default 336 px, patch 14, MLP projector and 64 text tokens, the image contributes 576 of the 640 tokens (90%). A 7B LLM then needs 320 MiB of fp16 KV cache and about 8.79 TFLOP of prefill, 10.2× the cost of the text alone. At 672 px the image costs 2,304 tokens (1,184 MiB, 34.7 TFLOP). At 1,344 px it costs 9,216 tokens and overflows a 4,096-token context; a 2×2 merge brings it back to 2,304. A Q-Former's 32 tokens cut the sequence to 96 (1.29 TFLOP), but fine detail must then survive a fixed bottleneck.

Benchmarks and the VQA score

VQAv2 (Goyal et al., CVPR 2017) balances each question with a similar image that has a different answer, so language priors alone score lower. Each question has 10 human answers, and the metric gives full credit when at least 3 humans agree:

Acc(a)=110∑k=110min⁡ ⁣(#{humans in the 9 others who said a}3, 1)\mathrm{Acc}(a)=\frac{1}{10}\sum_{k=1}^{10}\min\!\Big(\frac{\#\{\text{humans in the 9 others}\ \text{who said}\ a\}}{3},\,1\Big)
kk
leave one of the 10 annotators out, score against the other 9, and average over the 10 choices
aa
the model's answer after normalisation (lower-case, punctuation and articles removed, numbers as digits)
Model answerHumans agreeingPer-subset scoresVQA accuracy
red4min(4/3,1) or min(3/3,1) = 1 always1.000
dark red31 in 7 subsets, 2/3 in 30.900
maroon22/3 in 8 subsets, 1/3 in 20.600
brown11/3 in 9 subsets, 0 in 10.300
pink000.000
Worked example: 10 human answers were red ×4, dark red ×3, maroon ×2, brown ×1 (computed in Python)

Captioning is scored with CIDEr on COCO (Karpathy test split, 5,000 images). Newer suites test reasoning and reading: GQA, TextVQA, MMMU (college-level questions over diagrams and charts), and POPE (Li et al., 2023), which asks yes/no questions about objects to measure object hallucination.

python
1import torch
2from PIL import Image
3from transformers import AutoProcessor, LlavaForConditionalGeneration
4
5model_id = "llava-hf/llava-1.5-7b-hf"
6model = LlavaForConditionalGeneration.from_pretrained(
7 model_id, torch_dtype=torch.float16, device_map="auto")
8processor = AutoProcessor.from_pretrained(model_id)
9
10image = Image.open("street.jpg")
11prompt = "USER: <image>\nHow many cyclists are waiting at the light? Answer with a number. ASSISTANT:"
12inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device, torch.float16)
13print(inputs["input_ids"].shape) # the <image> placeholder expands to 576 visual positions
14out = model.generate(**inputs, max_new_tokens=10, do_sample=False)
15print(processor.decode(out[0], skip_special_tokens=True))
Asking LLaVA-1.5 a question with Hugging Face Transformers (not run here; needs a GPU with about 15 GB for the fp16 7B model)

Check yourself

A LLaVA-style model uses a ViT with patch size 14 at 448 × 448 and an MLP projector. How many visual tokens does the LLM see per image?

Check yourself

Why does Flamingo initialise the gate α of each new cross-attention layer to 0?

Remember

  • ✓Dual encoders (CLIP, SigLIP) align image and text embeddings for retrieval and zero-shot labels; they do not generate text.
  • ✓Flamingo: frozen vision encoder + Perceiver Resampler (64 tokens) + tanh-gated cross-attention layers in a frozen LM; few-shot in context.
  • ✓LLaVA: CLIP ViT-L/14 patch features → linear/MLP projector → LLM tokens, trained with next-token loss on GPT-4-generated visual instructions.
  • ✓Visual tokens N=(R/p)2N=(R/p)^2: 576 at 336 px with patch 14. Cost grows with resolution squared unless tokens are merged or resampled.
  • ✓VQAv2 accuracy is min(#agreeing humans / 3, 1), averaged over leave-one-out subsets; check hallucination separately (POPE).
← back to Deep Learning roadmap