Definition
Vision-language model (VLM) a model that takes images and text as input and either scores how well they match or generates text about the image. Three designs dominate. Dual encoders such as CLIP (Radford et al., ICML 2021) embed each modality separately. Cross-attention into a frozen language model, as in Flamingo (Alayrac et al., NeurIPS 2022). Projected visual tokens fed to a decoder-only LLM, as in LLaVA ("Visual Instruction Tuning", Liu, Li, Wu and Lee, NeurIPS 2023).
Tasks: image–text retrieval, zero-shot classification, captioning, visual question answering (VQA), and open-ended chat about images, charts and documents. The hard part is grounding: the text must be about what is actually in the pixels, including counts, positions and small text, and not about what is statistically likely.
Before 2021, captioning used a CNN encoder with an LSTM decoder (Show and Tell, Vinyals et al., 2015). VQA models attended over Faster R-CNN region features ("bottom-up and top-down attention", Anderson et al., CVPR 2018). Fusion encoders such as ViLBERT, LXMERT and UNITER ran one transformer over region features and words, with cross-modal attention in every layer. They were strong on VQA but could not retrieve at scale or write free text.
Three ways to connect vision to language
Dual encoders are covered in CLIP and SigLIP. They supply the vision encoder for most generative VLMs too, because contrastive pre-training on hundreds of millions of pairs gives patch features that already line up with language. For the general architecture patterns see Multimodal transformers and Cross-attention.
Flamingo adds trainable layers to a frozen Chinchilla LM (70B parameters in the largest, 80B-parameter Flamingo). Each new layer is gated so that the model starts as the unchanged language model:
- hidden states of the text tokens inside the frozen LM
- 64 visual tokens from the Perceiver Resampler (a set of learned latents that cross-attend to the image features)
- learned scalar per layer; tanh(0) = 0, so training begins from the LM's original behaviour
Trained on web pages with interleaved images and text, Flamingo does few-shot learning in context: a prompt with a few (image, answer) examples followed by a new image. BLIP and BLIP-2 (Li, Li, Savarese and Hoi, ICML 2023) take a similar route. BLIP-2's Q-Former compresses the image into 32 learned query tokens for a frozen LLM.
LLaVA: visual tokens plus instruction tuning
LLaVA uses the simplest connector. Patch features from a frozen CLIP ViT-L/14 are mapped by a projection into the LLM's word-embedding space and placed before the instruction. Training uses the ordinary language-modelling loss, on the answer tokens only:
- CLIP vision encoder; LLaVA takes the grid features of the second-to-last layer, not the [CLS] embedding
- one linear layer in LLaVA; a 2-layer MLP with GELU in LLaVA-1.5
- visual tokens: 256 at 224 px, 576 at 336 px with patch size 14
The data was the novelty. GPT-4, given only COCO captions and bounding boxes as text, wrote 158K instruction-following samples: conversations, detailed descriptions and multi-step reasoning about each image. Stage 1 trains only the projector on image–caption pairs, to align features. Stage 2 trains the projector and the LLM on the instructions. This recipe, visual instruction tuning, became the template for most open VLMs.
Reference configurations
- CLIP ViT-L/14@336px
- 400M image–text pairs; 76.2% zero-shot ImageNet top-1; 24 × 24 = 576 patch tokens of width 1024
- Flamingo-80B
- frozen NFNet-F6 vision encoder + frozen Chinchilla 70B LM; 64 visual tokens per image from the Perceiver Resampler
- BLIP-2
- frozen ViT-g image encoder + Q-Former with 32 queries + frozen OPT or Flan-T5 LLM
- LLaVA-1.5
- CLIP ViT-L/14 at 336 px + 2-layer MLP + Vicuna-7B/13B; 558K caption pairs (stage 1) and a 665K instruction mix (stage 2); about one day of training on one 8×A100 node; VQAv2 78.5 (7B) and 80.0 (13B)
Benchmarks and the VQA score
VQAv2 (Goyal et al., CVPR 2017) balances each question with a similar image that has a different answer, so language priors alone score lower. Each question has 10 human answers, and the metric gives full credit when at least 3 humans agree:
- leave one of the 10 annotators out, score against the other 9, and average over the 10 choices
- the model's answer after normalisation (lower-case, punctuation and articles removed, numbers as digits)
| Model answer | Humans agreeing | Per-subset scores | VQA accuracy |
|---|---|---|---|
| red | 4 | min(4/3,1) or min(3/3,1) = 1 always | 1.000 |
| dark red | 3 | 1 in 7 subsets, 2/3 in 3 | 0.900 |
| maroon | 2 | 2/3 in 8 subsets, 1/3 in 2 | 0.600 |
| brown | 1 | 1/3 in 9 subsets, 0 in 1 | 0.300 |
| pink | 0 | 0 | 0.000 |
Captioning is scored with CIDEr on COCO (Karpathy test split, 5,000 images). Newer suites test reasoning and reading: GQA, TextVQA, MMMU (college-level questions over diagrams and charts), and POPE (Li et al., 2023), which asks yes/no questions about objects to measure object hallucination.
1import torch2from PIL import Image3from transformers import AutoProcessor, LlavaForConditionalGeneration45model_id = "llava-hf/llava-1.5-7b-hf"6model = LlavaForConditionalGeneration.from_pretrained(7 model_id, torch_dtype=torch.float16, device_map="auto")8processor = AutoProcessor.from_pretrained(model_id)910image = Image.open("street.jpg")11prompt = "USER: <image>\nHow many cyclists are waiting at the light? Answer with a number. ASSISTANT:"12inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device, torch.float16)13print(inputs["input_ids"].shape) # the <image> placeholder expands to 576 visual positions14out = model.generate(**inputs, max_new_tokens=10, do_sample=False)15print(processor.decode(out[0], skip_special_tokens=True))Check yourself
A LLaVA-style model uses a ViT with patch size 14 at 448 × 448 and an MLP projector. How many visual tokens does the LLM see per image?
Check yourself
Why does Flamingo initialise the gate α of each new cross-attention layer to 0?
Remember
- ✓Dual encoders (CLIP, SigLIP) align image and text embeddings for retrieval and zero-shot labels; they do not generate text.
- ✓Flamingo: frozen vision encoder + Perceiver Resampler (64 tokens) + tanh-gated cross-attention layers in a frozen LM; few-shot in context.
- ✓LLaVA: CLIP ViT-L/14 patch features → linear/MLP projector → LLM tokens, trained with next-token loss on GPT-4-generated visual instructions.
- ✓Visual tokens : 576 at 336 px with patch 14. Cost grows with resolution squared unless tokens are merged or resampled.
- ✓VQAv2 accuracy is min(#agreeing humans / 3, 1), averaged over leave-one-out subsets; check hallucination separately (POPE).