# Multimodal LLMs

> A multimodal LLM turns each image into hundreds or thousands of visual tokens inside the prompt. The input scheme (fixed 336 px = 576 tokens, tiles, or native resolution) sets what detail survives, and what each request costs.

Source: https://www.mlfoundry.in/learn/multimodal-llms  
Section: Generative AI › Multimodal Generative AI  
Level: core · 7 min read · updated 2026-10-21

**Multimodal LLM (MLLM):** a large language model that accepts images (and in some systems video frames or audio) alongside text in the prompt, and answers in text. A vision encoder turns the image into a sequence of embeddings that the LLM reads like tokens. Landmarks: Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023), LLaVA ("Visual Instruction Tuning", Liu, Li, Wu and Lee, 2023) and Qwen2-VL (Wang et al., 2024); commercial models such as GPT-4V (2023) and Gemini (2023) brought the same interface to APIs.

For a practitioner an MLLM is a chat model whose messages can contain image parts. It reads receipts, charts, screenshots, scanned forms, slides and photos, and answers questions about them, extracts fields to JSON, or describes them. The tasks themselves are covered in **Image captioning** and **Visual question answering**; the encoder–projector–LLM architecture in **Vision-language models**. This article is about the part you control when you call one: how the image becomes tokens, what that costs, and where it fails.

## From pixels to tokens

The common recipe: a ViT pretrained with a contrastive loss (**CLIP** or **SigLIP**) cuts the image into patches, a small projector maps each patch feature to the LLM's embedding width, and the resulting vectors replace an `<image>` placeholder in the chat template. From there on the LLM treats them as prompt positions: they take context, compute and KV-cache memory like text tokens.

> Figure: One request, LLaVA-1.5 style. The message holds an image part and a text part. The image is resized to 336 px; a ViT-L/14 turns it into 24 × 24 = 576 patch features of width 1024; a two-layer MLP projects them to 4096, the LLM's width. The 576 visual embeddings sit between USER: and the question, and the decoder generates the answer as text. A one-line question carries 576 tokens of image.

$$
N_\text{fixed}=\Big(\frac{S}{p}\Big)^2=\Big(\frac{336}{14}\Big)^2=576,\qquad N_\text{tiles}=576\,(g_x g_y+1),\qquad N_\text{native}\approx\Big\lceil\frac{H}{28}\Big\rceil\Big\lceil\frac{W}{28}\Big\rceil\le N_\text{max}
$$

- $S,\ p$: encoder input size and patch size (LLaVA-1.5: 336 px, 14 px)
- $g_x g_y$: number of 336-px tiles in the chosen grid; the +1 is a downscaled thumbnail of the whole image
- $28$: Qwen2-VL: 14-px patches, and 2 × 2 neighbouring patches merged into one token
- $N_\text{max}$: a cap set through max_pixels; larger images are scaled down to fit it

The three schemes trade detail against tokens. **Fixed** resizes everything to one square: cheap and predictable, but a 12-megapixel photo is shrunk about 12×, and small print disappears. **Tiling** (AnyRes in LLaVA-NeXT, 2024) picks a grid of 336-px crops that fits the aspect ratio and adds a thumbnail for global context. **Native resolution** (Qwen2-VL's naive dynamic resolution) keeps the aspect ratio and makes the token count proportional to the pixel count, with a floor and a cap.

$$
\text{cost per request}=\big(N_\text{image}+N_\text{text}\big)\cdot\text{price}_\text{in}+N_\text{out}\cdot\text{price}_\text{out}
$$

- $N_\text{image}$: visual tokens, billed as input tokens by API providers

**Configurations and documented rules**

- **LLaVA-1.5 (2023):** CLIP ViT-L/14 at 336 px, 2-layer MLP projector, Vicuna LLM; 558K caption pairs to align the projector, then 665K visual-instruction examples
- **Qwen2-VL (2024):** native resolution, one token per 28 × 28 px; `min_pixels` / `max_pixels` set the range, and the README example uses 256·28·28 to 1280·28·28
- **OpenAI (GPT-4o, as documented in 2024):** low detail: 85 tokens. High detail: fit in 2048 × 2048, scale the short side to 768, then 85 + 170 per 512-px tile
- **Anthropic (Claude, as documented in 2024):** about width × height / 750 tokens; large images are downscaled first (long edge 1568 px)
- **Benchmarks:** MMMU (college-level multi-discipline), DocVQA and ChartQA (documents, charts), TextVQA (text in photos), POPE (object hallucination)

## The token budget of one image

> Figure: A 4032 × 3024 phone photo with 100 text tokens at USD 1 per million input tokens. Fixed: 576 visual tokens, USD 0.676 per 1,000 requests, and 12-px text in the photo becomes 1.0 px. AnyRes picks the 2 × 2 grid: 2,880 tokens, USD 2.98, 2.0 px. Native at a 1,280-token cap resizes to 1148 × 840: 1,230 tokens, USD 1.33, 3.4 px; raise the cap to 4,096 and it is 2044 × 1540, 4,015 tokens, 6.1 px. The image is 85–98% of the prompt in every case. Try the 512 × 512 icon: AnyRes still spends 2,880 tokens because it upscales to a 2 × 2 grid, while native uses 324.

## Worked example: one photo, four settings

4032 × 3024 photo, 100 text tokens, USD 1 per million input tokens (a round price for illustration). Computed in Python

| Scheme | Visual tokens | Prompt tokens | Image share | USD per 1,000 requests | 12-px text becomes |
| --- | --- | --- | --- | --- | --- |
| Fixed 336 px | 576 | 676 | 85.2% | 0.676 | 1.0 px |
| AnyRes 2 × 2 + thumbnail | 2,880 | 2,980 | 96.6% | 2.980 | 2.0 px |
| Native, cap 1,280 (1148 × 840) | 1,230 | 1,330 | 92.5% | 1.330 | 3.4 px |
| Native, cap 4,096 (2044 × 1540) | 4,015 | 4,115 | 97.6% | 4.115 | 6.1 px |

The 12-px line is the deciding number for documents. At 1–2 px no encoder can read it; the model will guess, and the guess will sound confident. Cropping the region of interest before sending it often beats raising the cap: a 600 × 400 crop of the receipt total costs a fraction of the full photo and keeps the text sharp.

Choosing an input scheme

| Scheme | Tokens per image | What survives | Good for |
| --- | --- | --- | --- |
| Fixed square | constant (576 at 336 px) | layout and objects; small text is lost | scene photos, classification, cheap batch captioning |
| Tiles + thumbnail | 576 × (tiles + 1), up to 2,880 here | local detail in each tile, global view in the thumbnail | documents and charts in models trained this way |
| Native resolution | ∝ pixels, between a floor and a cap | aspect ratio and detail up to the cap | screenshots, long receipts, mixed sizes; budget set per request |

## Uses and practice

- **Document extraction:** invoices, receipts, forms and ID cards to JSON fields, often replacing an OCR-plus-rules pipeline
- **Charts and slides:** reading values, trends and labels from plots and presentations; checkable against the source data
- **UI understanding:** screenshots for support tickets, accessibility descriptions, and computer-use agents that click on what they see
- **Visual inspection:** describing defects or damage in photos for triage, with a human making the final call

```python
import base64, math
from openai import OpenAI

def openai_image_tokens(w, h, detail="high"):
    """GPT-4o image-token rule as documented by OpenAI in 2024 (check current docs)."""
    if detail == "low":
        return 85
    s = min(1.0, 2048 / max(w, h)); w, h = w * s, h * s     # fit inside 2048 x 2048
    s = min(1.0, 768 / min(w, h));  w, h = w * s, h * s     # shortest side to 768
    return 85 + 170 * math.ceil(w / 512) * math.ceil(h / 512)

def qwen2vl_tokens(w, h, max_tokens=1280):
    """Qwen2-VL: one token per 28 x 28 pixels after rounding and capping."""
    F = 28
    hb, wb = round(h / F) * F, round(w / F) * F
    if hb * wb > max_tokens * F * F:
        beta = math.sqrt(h * w / (max_tokens * F * F))
        hb, wb = math.floor(h / beta / F) * F, math.floor(w / beta / F) * F
    return (hb // F) * (wb // F)

print(openai_image_tokens(4032, 3024), openai_image_tokens(4032, 3024, "low"), qwen2vl_tokens(4032, 3024))

b64 = base64.b64encode(open("receipt.jpg", "rb").read()).decode()
resp = OpenAI().chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}", "detail": "high"}},
        {"type": "text", "text": "Return the merchant, date and total as JSON."},
    ]}],
)
print(resp.usage.prompt_tokens)   # includes the image tokens
```

Estimate image tokens before sending (the two estimators run here: 765 / 85 / 1230 for the photo), then the request in the OpenAI chat format (not run here; model names change)

1. **Size images on purpose.** Downscale photos of scenes; crop or keep full resolution for documents, charts and UI screenshots. Check the provider's token rule, or measure `usage.prompt_tokens` on a sample.
2. **One image, one question.** With several images, label them in the text ("Image 1: invoice, Image 2: delivery note") and refer to them by label; models confuse unlabelled images.
3. **Ask for evidence.** For extraction, request the exact text span or a bounding region next to each value, and validate against a schema (**Structured output (JSON mode)**).
4. **Cache repeated images.** A fixed image prefix (a form template, a product photo) qualifies for **Prompt caching** like any prompt prefix.
5. **Treat text in images as untrusted input.** Instructions written in a screenshot or on a sign reach the model like typed text: this is **Prompt injection** through pixels.

> **Object hallucination and counting:** MLLMs describe objects that are typical for a scene but absent, which is what the POPE benchmark probes with yes/no questions ("Is there a fork?"). They also miscount beyond a handful of items and misread small or rotated text. For high-stakes extraction, compare against OCR output, ask the same question about a crop, or require the model to quote what it read. See **Hallucination** for the general case.

> **Visual tokens are prompt tokens:** Everything that is true of long prompts is true of images: latency to first token grows with them, they fill the context window, and they are billed. A chat history with ten screenshots at 1,000+ tokens each carries 10,000+ tokens per turn. Dropping old images from the history, or replacing them with their extracted text, is often the largest single saving.

**Question:** Under a LLaVA-NeXT-style AnyRes scheme, why can a 512 × 512 icon cost more tokens than a 1920 × 1080 screenshot needs under native resolution with a 1,280-token cap?

- A. Icons have more colours
- B. AnyRes chooses the grid that keeps the most pixels, and for a square image that is 2 × 2 tiles (upscaled) plus a thumbnail: 2,880 tokens
- C. Native resolution skips square images
- D. Tiles are always 1,024 tokens

**Answer:** B. AnyRes chooses the grid that keeps the most pixels, and for a square image that is 2 × 2 tiles (upscaled) plus a thumbnail: 2,880 tokens — The grid selection maximises preserved resolution, so a 512-px image is scaled up to 672 × 672 and cut into four tiles plus the thumbnail: $576\times5=2{,}880$ tokens. Native resolution rounds 512 to 504 and uses $18\times18=324$ tokens; the 1920 × 1080 screenshot at the 1,280 cap uses 1,222.

**Question:** A model misreads the total on photos of receipts but reads typed text perfectly. What is the most likely cause and fix?

- A. The LLM is too small; switch to a bigger LLM
- B. The image is downscaled until the digits are 1–3 px tall; crop the region or raise the pixel budget
- C. The temperature is too high
- D. Receipts need a different tokenizer

**Answer:** B. The image is downscaled until the digits are 1–3 px tall; crop the region or raise the pixel budget — A 4032 × 3024 photo resized to 336 px shrinks 12-px print to 1 px. The language model never saw the digits. A crop, a higher detail setting or a larger max_pixels gives the encoder enough pixels per character.

**Key points**

- An MLLM = vision encoder (CLIP/SigLIP ViT) + projector + LLM; the image becomes embeddings in the prompt.
- Fixed 336 px with 14-px patches gives $24^2=576$ tokens; tiling gives $576(g_xg_y+1)$; native resolution gives about one token per 28 × 28 px up to a cap.
- Visual tokens are billed, cached and attended like text tokens, and usually dominate the prompt (85–98% for one photo and a short question).
- Detail is set by pixels per character: crop documents, downscale scenes.
- Failure modes: object hallucination (POPE), counting, small text, and instructions hidden in images.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
