# Vision-language models

> Vision-language models join an image encoder to language: dual encoders (CLIP) align embeddings, Flamingo cross-attends into a frozen LM, and LLaVA projects 576 CLIP patch tokens into an LLM tuned on visual instructions.

Source: https://www.mlfoundry.in/learn/vision-language-models  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Vision-language model (VLM):** a model that takes images and text as input and either scores how well they match or generates text about the image. Three designs dominate. **Dual encoders** such as **CLIP** (Radford et al., ICML 2021) embed each modality separately. **Cross-attention into a frozen language model**, as in Flamingo (Alayrac et al., NeurIPS 2022). **Projected visual tokens** fed to a decoder-only LLM, as in LLaVA ("Visual Instruction Tuning", Liu, Li, Wu and Lee, NeurIPS 2023).

Tasks: image–text retrieval, zero-shot classification, captioning, visual question answering (VQA), and open-ended chat about images, charts and documents. The hard part is grounding: the text must be about what is actually in the pixels, including counts, positions and small text, and not about what is statistically likely.

Before 2021, captioning used a CNN encoder with an LSTM decoder (Show and Tell, Vinyals et al., 2015). VQA models attended over Faster R-CNN region features ("bottom-up and top-down attention", Anderson et al., CVPR 2018). **Fusion encoders** such as ViLBERT, LXMERT and UNITER ran one transformer over region features and words, with cross-modal attention in every layer. They were strong on VQA but could not retrieve at scale or write free text.

## Three ways to connect vision to language

> Figure: (a) A dual encoder scores image–caption pairs through a cosine-similarity matrix. It is cheap for retrieval but cannot generate text. (b) Flamingo keeps both the vision encoder and the language model frozen. A Perceiver Resampler turns the image features into 64 tokens, and new tanh-gated cross-attention layers, inserted between the LM blocks, read them. (c) LLaVA-1.5 projects the 576 patch features of CLIP ViT-L/14 at 336 px (width 1024) with a 2-layer MLP to the LLM width of 4096. It then simply prepends them to the text tokens.

Dual encoders are covered in **CLIP** and **SigLIP**. They supply the vision encoder for most generative VLMs too, because contrastive pre-training on hundreds of millions of pairs gives patch features that already line up with language. For the general architecture patterns see **Multimodal transformers** and **Cross-attention**.

Flamingo adds trainable layers to a frozen Chinchilla LM (70B parameters in the largest, 80B-parameter Flamingo). Each new layer is gated so that the model starts as the unchanged language model:

$$
x\leftarrow x+\tanh(\alpha)\cdot\mathrm{XAttn}(x,\,z),\qquad \alpha=0\ \text{at initialisation}
$$

- $x$: hidden states of the text tokens inside the frozen LM
- $z$: 64 visual tokens from the Perceiver Resampler (a set of learned latents that cross-attend to the image features)
- $\alpha$: learned scalar per layer; tanh(0) = 0, so training begins from the LM's original behaviour

Trained on web pages with interleaved images and text, Flamingo does few-shot learning in context: a prompt with a few (image, answer) examples followed by a new image. **BLIP** and BLIP-2 (Li, Li, Savarese and Hoi, ICML 2023) take a similar route. BLIP-2's Q-Former compresses the image into 32 learned query tokens for a frozen LLM.

## LLaVA: visual tokens plus instruction tuning

LLaVA uses the simplest connector. Patch features $Z_v$ from a frozen CLIP ViT-L/14 are mapped by a projection $W$ into the LLM's word-embedding space and placed before the instruction. Training uses the ordinary language-modelling loss, on the answer tokens only:

$$
H_v=W\cdot Z_v,\quad Z_v=g(X_v),\qquad \mathcal L=-\sum_{t\in\text{answer}}\log p_\theta\big(x_t\mid H_v,\,X_{\text{instruct}},\,x_{<t}\big)
$$

- $g$: CLIP vision encoder; LLaVA takes the grid features of the second-to-last layer, not the [CLS] embedding
- $W$: one linear layer in LLaVA; a 2-layer MLP with GELU in LLaVA-1.5
- $H_v$: $N=(R/p)^2$ visual tokens: 256 at 224 px, 576 at 336 px with patch size 14

The data was the novelty. GPT-4, given only COCO captions and bounding boxes as text, wrote 158K instruction-following samples: conversations, detailed descriptions and multi-step reasoning about each image. Stage 1 trains only the projector on image–caption pairs, to align features. Stage 2 trains the projector and the LLM on the instructions. This recipe, **visual instruction tuning**, became the template for most open VLMs.

**Reference configurations**

- **CLIP ViT-L/14@336px:** 400M image–text pairs; 76.2% zero-shot ImageNet top-1; 24 × 24 = 576 patch tokens of width 1024
- **Flamingo-80B:** frozen NFNet-F6 vision encoder + frozen Chinchilla 70B LM; 64 visual tokens per image from the Perceiver Resampler
- **BLIP-2:** frozen ViT-g image encoder + Q-Former with 32 queries + frozen OPT or Flan-T5 LLM
- **LLaVA-1.5:** CLIP ViT-L/14 at 336 px + 2-layer MLP + Vicuna-7B/13B; 558K caption pairs (stage 1) and a 665K instruction mix (stage 2); about one day of training on one 8×A100 node; VQAv2 78.5 (7B) and 80.0 (13B)

> Figure: The cost of visual tokens in a LLaVA-style model. At the default 336 px, patch 14, MLP projector and 64 text tokens, the image contributes 576 of the 640 tokens (90%). A 7B LLM then needs 320 MiB of fp16 KV cache and about 8.79 TFLOP of prefill, 10.2× the cost of the text alone. At 672 px the image costs 2,304 tokens (1,184 MiB, 34.7 TFLOP). At 1,344 px it costs 9,216 tokens and overflows a 4,096-token context; a 2×2 merge brings it back to 2,304. A Q-Former's 32 tokens cut the sequence to 96 (1.29 TFLOP), but fine detail must then survive a fixed bottleneck.

## Benchmarks and the VQA score

VQAv2 (Goyal et al., CVPR 2017) balances each question with a similar image that has a different answer, so language priors alone score lower. Each question has 10 human answers, and the metric gives full credit when at least 3 humans agree:

$$
\mathrm{Acc}(a)=\frac{1}{10}\sum_{k=1}^{10}\min\!\Big(\frac{\#\{\text{humans in the 9 others}\ \text{who said}\ a\}}{3},\,1\Big)
$$

- $k$: leave one of the 10 annotators out, score against the other 9, and average over the 10 choices
- $a$: the model's answer after normalisation (lower-case, punctuation and articles removed, numbers as digits)

Worked example: 10 human answers were red ×4, dark red ×3, maroon ×2, brown ×1 (computed in Python)

| Model answer | Humans agreeing | Per-subset scores | VQA accuracy |
| --- | --- | --- | --- |
| red | 4 | min(4/3,1) or min(3/3,1) = 1 always | 1.000 |
| dark red | 3 | 1 in 7 subsets, 2/3 in 3 | 0.900 |
| maroon | 2 | 2/3 in 8 subsets, 1/3 in 2 | 0.600 |
| brown | 1 | 1/3 in 9 subsets, 0 in 1 | 0.300 |
| pink | 0 | 0 | 0.000 |

Captioning is scored with CIDEr on COCO (Karpathy test split, 5,000 images). Newer suites test reasoning and reading: GQA, TextVQA, MMMU (college-level questions over diagrams and charts), and POPE (Li et al., 2023), which asks yes/no questions about objects to measure **object hallucination**.

```python
import torch
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration

model_id = "llava-hf/llava-1.5-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("street.jpg")
prompt = "USER: <image>\nHow many cyclists are waiting at the light? Answer with a number. ASSISTANT:"
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device, torch.float16)
print(inputs["input_ids"].shape)     # the <image> placeholder expands to 576 visual positions
out = model.generate(**inputs, max_new_tokens=10, do_sample=False)
print(processor.decode(out[0], skip_special_tokens=True))
```

Asking LLaVA-1.5 a question with Hugging Face Transformers (not run here; needs a GPU with about 15 GB for the fp16 7B model)

> **Fluent is not grounded:** VLMs inherit the language model's priors. Asked "what colour is the fire hydrant?", a model may answer "red" when no hydrant is visible. Counting, left/right relations and small text are failure points, especially at 336 px where a 14-pixel patch can hold a whole word. Test with POPE-style negative questions, and raise the resolution (tiling) before blaming the LLM. See **Hallucination detection**.

> **The connector decides the cost:** Every visual token is a full LLM token: it is processed by every layer and kept in the KV cache. An MLP projector keeps all spatial detail but makes cost grow with resolution squared. A resampler or Q-Former fixes the cost at 32–64 tokens but forces a bottleneck. Modern models mix both: high-resolution tiles with 2×2 token merging.

**Question:** A LLaVA-style model uses a ViT with patch size 14 at 448 × 448 and an MLP projector. How many visual tokens does the LLM see per image?

- A. 256
- B. 576
- C. 1,024
- D. 2,048

**Answer:** C. 1,024 — $448/14=32$, so the grid is $32\times32=1{,}024$ patches, and an MLP projector keeps one token per patch. At 336 px it would be $24^2=576$.

**Question:** Why does Flamingo initialise the gate α of each new cross-attention layer to 0?

- A. So that at the start of training the frozen LM behaves exactly as before, and visual input is blended in gradually
- B. To stop gradients from reaching the vision encoder
- C. Because tanh is undefined for non-zero arguments
- D. To make the Perceiver Resampler output exactly 64 tokens

**Answer:** A. So that at the start of training the frozen LM behaves exactly as before, and visual input is blended in gradually — tanh(0) = 0, so the residual branch contributes nothing at initialisation and the pretrained LM is untouched. Randomly initialised cross-attention added at full strength would disrupt the frozen LM's representations at the start of training.

**Key points**

- Dual encoders (CLIP, SigLIP) align image and text embeddings for retrieval and zero-shot labels; they do not generate text.
- Flamingo: frozen vision encoder + Perceiver Resampler (64 tokens) + tanh-gated cross-attention layers in a frozen LM; few-shot in context.
- LLaVA: CLIP ViT-L/14 patch features → linear/MLP projector → LLM tokens, trained with next-token loss on GPT-4-generated visual instructions.
- Visual tokens $N=(R/p)^2$: 576 at 336 px with patch 14. Cost grows with resolution squared unless tokens are merged or resampled.
- VQAv2 accuracy is min(#agreeing humans / 3, 1), averaged over leave-one-out subsets; check hallucination separately (POPE).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
