Multimodal LLMs

A multimodal LLM turns each image into hundreds or thousands of visual tokens inside the prompt. The input scheme (fixed 336 px = 576 tokens, tiles, or native resolution) sets what detail survives, and what each request costs.

7 min read726 wordscore0 viewsupdated 2026-10-21

Definition

Multimodal LLM (MLLM) a large language model that accepts images (and in some systems video frames or audio) alongside text in the prompt, and answers in text. A vision encoder turns the image into a sequence of embeddings that the LLM reads like tokens. Landmarks: Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023), LLaVA ("Visual Instruction Tuning", Liu, Li, Wu and Lee, 2023) and Qwen2-VL (Wang et al., 2024); commercial models such as GPT-4V (2023) and Gemini (2023) brought the same interface to APIs.

For a practitioner an MLLM is a chat model whose messages can contain image parts. It reads receipts, charts, screenshots, scanned forms, slides and photos, and answers questions about them, extracts fields to JSON, or describes them. The tasks themselves are covered in Image captioning and Visual question answering; the encoder–projector–LLM architecture in Vision-language models. This article is about the part you control when you call one: how the image becomes tokens, what that costs, and where it fails.

From pixels to tokens

The common recipe: a ViT pretrained with a contrastive loss (CLIP or SigLIP) cuts the image into patches, a small projector maps each patch feature to the LLM's embedding width, and the resulting vectors replace an <image> placeholder in the chat template. From there on the LLM treats them as prompt positions: they take context, compute and KV-cache memory like text tokens.

a multimodal chat request, LLaVA-style: the image becomes tokens in the prompt{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}, {"type": "text", "text": "What is the total on this receipt?"}]}image part → vision pathtext part → tokenizerthe chat template puts an<image> placeholder in the textreceipt800×2400resize / tile336² per viewViT-L/1424×24 × 1024MLP proj.1024 → 4096576 tokensper viewViT usually frozen; projector trained first (stage 1)input sequence to the LLM (width 4096)<s>USER:<image> → 576 visual embeddingsWhatisthetotal?ASSISTANT:decoder-only LLM (e.g. 7B): causal attention over text and visual positions alike"The total is $23.40." (generated one text token at a time)the 576 visual tokens are billed and cached like prompt tokens; they dominate a short question
fig.One request, LLaVA-1.5 style. The message holds an image part and a text part. The image is resized to 336 px; a ViT-L/14 turns it into 24 × 24 = 576 patch features of width 1024; a two-layer MLP projects them to 4096, the LLM's width. The 576 visual embeddings sit between USER: and the question, and the decoder generates the answer as text. A one-line question carries 576 tokens of image.
Nfixed=(Sp)2=(33614)2=576,Ntiles=576 (gxgy+1),Nnative≈⌈H28⌉⌈W28⌉≤NmaxN_\text{fixed}=\Big(\frac{S}{p}\Big)^2=\Big(\frac{336}{14}\Big)^2=576,\qquad N_\text{tiles}=576\,(g_x g_y+1),\qquad N_\text{native}\approx\Big\lceil\frac{H}{28}\Big\rceil\Big\lceil\frac{W}{28}\Big\rceil\le N_\text{max}
S, pS,\ p
encoder input size and patch size (LLaVA-1.5: 336 px, 14 px)
gxgyg_x g_y
number of 336-px tiles in the chosen grid; the +1 is a downscaled thumbnail of the whole image
2828
Qwen2-VL: 14-px patches, and 2 × 2 neighbouring patches merged into one token
NmaxN_\text{max}
a cap set through max_pixels; larger images are scaled down to fit it

The three schemes trade detail against tokens. Fixed resizes everything to one square: cheap and predictable, but a 12-megapixel photo is shrunk about 12×, and small print disappears. Tiling (AnyRes in LLaVA-NeXT, 2024) picks a grid of 336-px crops that fits the aspect ratio and adds a thumbnail for global context. Native resolution (Qwen2-VL's naive dynamic resolution) keeps the aspect ratio and makes the token count proportional to the pixel count, with a floor and a cap.

cost per request=(Nimage+Ntext)⋅pricein+Nout⋅priceout\text{cost per request}=\big(N_\text{image}+N_\text{text}\big)\cdot\text{price}_\text{in}+N_\text{out}\cdot\text{price}_\text{out}
NimageN_\text{image}
visual tokens, billed as input tokens by API providers

Configurations and documented rules

LLaVA-1.5 (2023)
CLIP ViT-L/14 at 336 px, 2-layer MLP projector, Vicuna LLM; 558K caption pairs to align the projector, then 665K visual-instruction examples
Qwen2-VL (2024)
native resolution, one token per 28 × 28 px; min_pixels / max_pixels set the range, and the README example uses 256·28·28 to 1280·28·28
OpenAI (GPT-4o, as documented in 2024)
low detail: 85 tokens. High detail: fit in 2048 × 2048, scale the short side to 768, then 85 + 170 per 512-px tile
Anthropic (Claude, as documented in 2024)
about width × height / 750 tokens; large images are downscaled first (long edge 1568 px)
Benchmarks
MMMU (college-level multi-discipline), DocVQA and ChartQA (documents, charts), TextVQA (text in photos), POPE (object hallucination)

The token budget of one image

input image 4032×3024 (12.2 MP) + 100 text tokens · how many visual tokens?grey = canvas the encoder sees (padding included), tinted = the image, lines = tiles or token gridfixed 336 px (LLaVA-1.5)pad to squareresize to 336 pxAnyRes tiles (LLaVA-NeXT)+ thumbnail2×2 tiles of 336 px + 1thumbnailnative res. (Qwen2-VL)1148×840 pxone token per 28×28fixed: 576 + 100$0.676/1k req · 12 px → 1.0 pxanyres: 2880 + 100$2.980/1k req · 12 px → 2.0 pxnative: 1230 + 100$1.330/1k req · 12 px → 3.4 pximage share of the prompt: fixed 85.2%, anyres 96.6%, native 92.5% · grey bar = text
image
fig.A 4032 × 3024 phone photo with 100 text tokens at USD 1 per million input tokens. Fixed: 576 visual tokens, USD 0.676 per 1,000 requests, and 12-px text in the photo becomes 1.0 px. AnyRes picks the 2 × 2 grid: 2,880 tokens, USD 2.98, 2.0 px. Native at a 1,280-token cap resizes to 1148 × 840: 1,230 tokens, USD 1.33, 3.4 px; raise the cap to 4,096 and it is 2044 × 1540, 4,015 tokens, 6.1 px. The image is 85–98% of the prompt in every case. Try the 512 × 512 icon: AnyRes still spends 2,880 tokens because it upscales to a 2 × 2 grid, while native uses 324.

Worked example: one photo, four settings

SchemeVisual tokensPrompt tokensImage shareUSD per 1,000 requests12-px text becomes
Fixed 336 px57667685.2%0.6761.0 px
AnyRes 2 × 2 + thumbnail2,8802,98096.6%2.9802.0 px
Native, cap 1,280 (1148 × 840)1,2301,33092.5%1.3303.4 px
Native, cap 4,096 (2044 × 1540)4,0154,11597.6%4.1156.1 px
4032 × 3024 photo, 100 text tokens, USD 1 per million input tokens (a round price for illustration). Computed in Python

The 12-px line is the deciding number for documents. At 1–2 px no encoder can read it; the model will guess, and the guess will sound confident. Cropping the region of interest before sending it often beats raising the cap: a 600 × 400 crop of the receipt total costs a fraction of the full photo and keeps the text sharp.

SchemeTokens per imageWhat survivesGood for
Fixed squareconstant (576 at 336 px)layout and objects; small text is lostscene photos, classification, cheap batch captioning
Tiles + thumbnail576 × (tiles + 1), up to 2,880 herelocal detail in each tile, global view in the thumbnaildocuments and charts in models trained this way
Native resolution∝ pixels, between a floor and a capaspect ratio and detail up to the capscreenshots, long receipts, mixed sizes; budget set per request
Choosing an input scheme

Uses and practice

Where it's used

Document extraction

invoices, receipts, forms and ID cards to JSON fields, often replacing an OCR-plus-rules pipeline

Charts and slides

reading values, trends and labels from plots and presentations; checkable against the source data

UI understanding

screenshots for support tickets, accessibility descriptions, and computer-use agents that click on what they see

Visual inspection

describing defects or damage in photos for triage, with a human making the final call

python
1import base64, math
2from openai import OpenAI
3
4def openai_image_tokens(w, h, detail="high"):
5 """GPT-4o image-token rule as documented by OpenAI in 2024 (check current docs)."""
6 if detail == "low":
7 return 85
8 s = min(1.0, 2048 / max(w, h)); w, h = w * s, h * s # fit inside 2048 x 2048
9 s = min(1.0, 768 / min(w, h)); w, h = w * s, h * s # shortest side to 768
10 return 85 + 170 * math.ceil(w / 512) * math.ceil(h / 512)
11
12def qwen2vl_tokens(w, h, max_tokens=1280):
13 """Qwen2-VL: one token per 28 x 28 pixels after rounding and capping."""
14 F = 28
15 hb, wb = round(h / F) * F, round(w / F) * F
16 if hb * wb > max_tokens * F * F:
17 beta = math.sqrt(h * w / (max_tokens * F * F))
18 hb, wb = math.floor(h / beta / F) * F, math.floor(w / beta / F) * F
19 return (hb // F) * (wb // F)
20
21print(openai_image_tokens(4032, 3024), openai_image_tokens(4032, 3024, "low"), qwen2vl_tokens(4032, 3024))
22
23b64 = base64.b64encode(open("receipt.jpg", "rb").read()).decode()
24resp = OpenAI().chat.completions.create(
25 model="gpt-4o",
26 messages=[{"role": "user", "content": [
27 {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}", "detail": "high"}},
28 {"type": "text", "text": "Return the merchant, date and total as JSON."},
29 ]}],
30)
31print(resp.usage.prompt_tokens) # includes the image tokens
Estimate image tokens before sending (the two estimators run here: 765 / 85 / 1230 for the photo), then the request in the OpenAI chat format (not run here; model names change)
  1. 1Size images on purpose. Downscale photos of scenes; crop or keep full resolution for documents, charts and UI screenshots. Check the provider's token rule, or measure usage.prompt_tokens on a sample.
  2. 2One image, one question. With several images, label them in the text ("Image 1: invoice, Image 2: delivery note") and refer to them by label; models confuse unlabelled images.
  3. 3Ask for evidence. For extraction, request the exact text span or a bounding region next to each value, and validate against a schema (Structured output (JSON mode)).
  4. 4Cache repeated images. A fixed image prefix (a form template, a product photo) qualifies for Prompt caching like any prompt prefix.
  5. 5Treat text in images as untrusted input. Instructions written in a screenshot or on a sign reach the model like typed text: this is Prompt injection through pixels.

Check yourself

Under a LLaVA-NeXT-style AnyRes scheme, why can a 512 × 512 icon cost more tokens than a 1920 × 1080 screenshot needs under native resolution with a 1,280-token cap?

Check yourself

A model misreads the total on photos of receipts but reads typed text perfectly. What is the most likely cause and fix?

Remember

  • ✓An MLLM = vision encoder (CLIP/SigLIP ViT) + projector + LLM; the image becomes embeddings in the prompt.
  • ✓Fixed 336 px with 14-px patches gives 242=57624^2=576 tokens; tiling gives 576(gxgy+1)576(g_xg_y+1); native resolution gives about one token per 28 × 28 px up to a cap.
  • ✓Visual tokens are billed, cached and attended like text tokens, and usually dominate the prompt (85–98% for one photo and a short question).
  • ✓Detail is set by pixels per character: crop documents, downscale scenes.
  • ✓Failure modes: object hallucination (POPE), counting, small text, and instructions hidden in images.
← back to Generative AI roadmap