Definition
Multimodal LLM (MLLM) a large language model that accepts images (and in some systems video frames or audio) alongside text in the prompt, and answers in text. A vision encoder turns the image into a sequence of embeddings that the LLM reads like tokens. Landmarks: Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023), LLaVA ("Visual Instruction Tuning", Liu, Li, Wu and Lee, 2023) and Qwen2-VL (Wang et al., 2024); commercial models such as GPT-4V (2023) and Gemini (2023) brought the same interface to APIs.
For a practitioner an MLLM is a chat model whose messages can contain image parts. It reads receipts, charts, screenshots, scanned forms, slides and photos, and answers questions about them, extracts fields to JSON, or describes them. The tasks themselves are covered in Image captioning and Visual question answering; the encoder–projector–LLM architecture in Vision-language models. This article is about the part you control when you call one: how the image becomes tokens, what that costs, and where it fails.
From pixels to tokens
The common recipe: a ViT pretrained with a contrastive loss (CLIP or SigLIP) cuts the image into patches, a small projector maps each patch feature to the LLM's embedding width, and the resulting vectors replace an <image> placeholder in the chat template. From there on the LLM treats them as prompt positions: they take context, compute and KV-cache memory like text tokens.
- encoder input size and patch size (LLaVA-1.5: 336 px, 14 px)
- number of 336-px tiles in the chosen grid; the +1 is a downscaled thumbnail of the whole image
- Qwen2-VL: 14-px patches, and 2 × 2 neighbouring patches merged into one token
- a cap set through max_pixels; larger images are scaled down to fit it
The three schemes trade detail against tokens. Fixed resizes everything to one square: cheap and predictable, but a 12-megapixel photo is shrunk about 12×, and small print disappears. Tiling (AnyRes in LLaVA-NeXT, 2024) picks a grid of 336-px crops that fits the aspect ratio and adds a thumbnail for global context. Native resolution (Qwen2-VL's naive dynamic resolution) keeps the aspect ratio and makes the token count proportional to the pixel count, with a floor and a cap.
- visual tokens, billed as input tokens by API providers
Configurations and documented rules
- LLaVA-1.5 (2023)
- CLIP ViT-L/14 at 336 px, 2-layer MLP projector, Vicuna LLM; 558K caption pairs to align the projector, then 665K visual-instruction examples
- Qwen2-VL (2024)
- native resolution, one token per 28 × 28 px;
min_pixels/max_pixelsset the range, and the README example uses 256·28·28 to 1280·28·28 - OpenAI (GPT-4o, as documented in 2024)
- low detail: 85 tokens. High detail: fit in 2048 × 2048, scale the short side to 768, then 85 + 170 per 512-px tile
- Anthropic (Claude, as documented in 2024)
- about width × height / 750 tokens; large images are downscaled first (long edge 1568 px)
- Benchmarks
- MMMU (college-level multi-discipline), DocVQA and ChartQA (documents, charts), TextVQA (text in photos), POPE (object hallucination)
The token budget of one image
Worked example: one photo, four settings
| Scheme | Visual tokens | Prompt tokens | Image share | USD per 1,000 requests | 12-px text becomes |
|---|---|---|---|---|---|
| Fixed 336 px | 576 | 676 | 85.2% | 0.676 | 1.0 px |
| AnyRes 2 × 2 + thumbnail | 2,880 | 2,980 | 96.6% | 2.980 | 2.0 px |
| Native, cap 1,280 (1148 × 840) | 1,230 | 1,330 | 92.5% | 1.330 | 3.4 px |
| Native, cap 4,096 (2044 × 1540) | 4,015 | 4,115 | 97.6% | 4.115 | 6.1 px |
The 12-px line is the deciding number for documents. At 1–2 px no encoder can read it; the model will guess, and the guess will sound confident. Cropping the region of interest before sending it often beats raising the cap: a 600 × 400 crop of the receipt total costs a fraction of the full photo and keeps the text sharp.
| Scheme | Tokens per image | What survives | Good for |
|---|---|---|---|
| Fixed square | constant (576 at 336 px) | layout and objects; small text is lost | scene photos, classification, cheap batch captioning |
| Tiles + thumbnail | 576 × (tiles + 1), up to 2,880 here | local detail in each tile, global view in the thumbnail | documents and charts in models trained this way |
| Native resolution | ∝ pixels, between a floor and a cap | aspect ratio and detail up to the cap | screenshots, long receipts, mixed sizes; budget set per request |
Uses and practice
Where it's used
Document extraction
invoices, receipts, forms and ID cards to JSON fields, often replacing an OCR-plus-rules pipeline
Charts and slides
reading values, trends and labels from plots and presentations; checkable against the source data
UI understanding
screenshots for support tickets, accessibility descriptions, and computer-use agents that click on what they see
Visual inspection
describing defects or damage in photos for triage, with a human making the final call
1import base64, math2from openai import OpenAI34def openai_image_tokens(w, h, detail="high"):5 """GPT-4o image-token rule as documented by OpenAI in 2024 (check current docs)."""6 if detail == "low":7 return 858 s = min(1.0, 2048 / max(w, h)); w, h = w * s, h * s # fit inside 2048 x 20489 s = min(1.0, 768 / min(w, h)); w, h = w * s, h * s # shortest side to 76810 return 85 + 170 * math.ceil(w / 512) * math.ceil(h / 512)1112def qwen2vl_tokens(w, h, max_tokens=1280):13 """Qwen2-VL: one token per 28 x 28 pixels after rounding and capping."""14 F = 2815 hb, wb = round(h / F) * F, round(w / F) * F16 if hb * wb > max_tokens * F * F:17 beta = math.sqrt(h * w / (max_tokens * F * F))18 hb, wb = math.floor(h / beta / F) * F, math.floor(w / beta / F) * F19 return (hb // F) * (wb // F)2021print(openai_image_tokens(4032, 3024), openai_image_tokens(4032, 3024, "low"), qwen2vl_tokens(4032, 3024))2223b64 = base64.b64encode(open("receipt.jpg", "rb").read()).decode()24resp = OpenAI().chat.completions.create(25 model="gpt-4o",26 messages=[{"role": "user", "content": [27 {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}", "detail": "high"}},28 {"type": "text", "text": "Return the merchant, date and total as JSON."},29 ]}],30)31print(resp.usage.prompt_tokens) # includes the image tokens- 1Size images on purpose. Downscale photos of scenes; crop or keep full resolution for documents, charts and UI screenshots. Check the provider's token rule, or measure
usage.prompt_tokenson a sample. - 2One image, one question. With several images, label them in the text ("Image 1: invoice, Image 2: delivery note") and refer to them by label; models confuse unlabelled images.
- 3Ask for evidence. For extraction, request the exact text span or a bounding region next to each value, and validate against a schema (Structured output (JSON mode)).
- 4Cache repeated images. A fixed image prefix (a form template, a product photo) qualifies for Prompt caching like any prompt prefix.
- 5Treat text in images as untrusted input. Instructions written in a screenshot or on a sign reach the model like typed text: this is Prompt injection through pixels.
Check yourself
Under a LLaVA-NeXT-style AnyRes scheme, why can a 512 × 512 icon cost more tokens than a 1920 × 1080 screenshot needs under native resolution with a 1,280-token cap?
Check yourself
A model misreads the total on photos of receipts but reads typed text perfectly. What is the most likely cause and fix?
Remember
- ✓An MLLM = vision encoder (CLIP/SigLIP ViT) + projector + LLM; the image becomes embeddings in the prompt.
- ✓Fixed 336 px with 14-px patches gives tokens; tiling gives ; native resolution gives about one token per 28 × 28 px up to a cap.
- ✓Visual tokens are billed, cached and attended like text tokens, and usually dominate the prompt (85–98% for one photo and a short question).
- ✓Detail is set by pixels per character: crop documents, downscale scenes.
- ✓Failure modes: object hallucination (POPE), counting, small text, and instructions hidden in images.