BLIP

BLIP pre-trains one image–text model for understanding and generation (ITC, ITM and captioning losses on shared weights), then cleans its web data with CapFilt: a captioner adds captions and a filter drops mismatched texts.

8 min read857 wordsadvanced0 viewsupdated 2026-10-15

Definition

BLIP (Bootstrapping Language-Image Pre-training) is a vision–language model that is pre-trained with three objectives on shared weights, so one model can retrieve, classify image–text pairs and write captions. It also improves its own training data by generating and filtering captions. Introduced by Junnan Li, Dongxu Li, Caiming Xiong and Steven Hoi (Salesforce Research), "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation", ICML 2022.

Two problems motivated it. First, models were either encoders or encoder–decoders. Dual encoders such as CLIP are good at retrieval but cannot write text. Fused encoders (e.g. ALBEF) are good at matching and VQA but are not trained to generate. Second, web alt-text is noisy. Many captions describe the page, not the picture ("free shipping on all orders"), and scaling up such data gives diminishing returns.

Multimodal mixture of encoder–decoder (MED)

A ViT image encoder (ViT-B/16 or ViT-L/16, initialised from ImageNet) turns the image into a [CLS] embedding and patch features. A BERT-base text Transformer runs in three modes over the same caption:

  1. 1Unimodal text encoder. Bidirectional self-attention, no access to the image. Its [CLS] embedding is contrasted with the image [CLS] embedding (ITC).
  2. 2Image-grounded text encoder. A cross-attention layer between self-attention and feed-forward in every block attends to the patch features. An [Encode] token's output feeds a binary head: does this text match this image (ITM)?
  3. 3Image-grounded text decoder. The same blocks with causal self-attention and a [Decode] start token, trained to generate the caption (LM).
224 × 224 imageimage encoderViT-B/16[CLS] + 196patch featuresimage features (K, V)ITCimg [CLS] vs txt [CLS]feed forwardbi self-attention× 12[CLS] a dog catches…text encoderITMmatch / no matchfeed forwardcross-attentionbi self-attention× 12[Encode] a dog catches…image-grounded encoderLMnext-token CEfeed forwardcross-attentioncausal self-attention× 12[Decode] a dog catches…image-grounded decodersame colour = shared weights; the decoder differs from the grounded encoder only in self-attentionCapFilt: bootstrapping the web dataT_w "blue toy, free ship"filter (ITM)p(match) 0.08 → droppedI_wcaptionerT_s "a dog catches…"filter: p(match) 0.93 → keptnew datasetkept T_w, T_s+ human pairs→ new BLIPcaptioner = image-grounded decoder, filter = image-grounded encoder; both start from the pre-trainedmodel and are fine-tuned separately on COCO; captions use nucleus sampling, not beam search.
fig.Top: the three text modes over one ViT. Cross-attention (orange) is shared between the grounded encoder and the decoder, and feed-forward layers (violet) are shared by all three. Only the self-attention differs between encoding (bidirectional, blue) and decoding (causal, red). The unimodal encoder is the grounded encoder with cross-attention skipped. Bottom: CapFilt. The filter gives the web text "blue toy, free ship" p(match) = 0.08 and drops it. The captioner's "a dog catches…" gets 0.93 and is kept. Kept texts plus human-annotated pairs pre-train a new model.

The encoder and decoder share all parameters except the self-attention layers. The authors found that self-attention is where encoding and decoding conflict, and that sharing everything else gives better results than separate models with fewer parameters. Each pair costs one forward pass of the image encoder and three of the text Transformer.

L=12 E[H(yi2t,pi2t)+H(yt2i,pt2i)]⏟ITC  +  E[H(yitm,pitm(I,T))]⏟ITM  +  E[−∑tlog⁡Pθ(wt∣w<t,I)]⏟LM\mathcal L=\underbrace{\tfrac12\,\mathbb E\big[H(y^{i2t},p^{i2t})+H(y^{t2i},p^{t2i})\big]}_{\text{ITC}}\;+\;\underbrace{\mathbb E\big[H(y^{itm},p^{itm}(I,T))\big]}_{\text{ITM}}\;+\;\underbrace{\mathbb E\Big[-\textstyle\sum_{t}\log P_\theta(w_t\mid w_{<t},I)\Big]}_{\text{LM}}
pi2t, pt2ip^{i2t},\,p^{t2i}
softmax over the batch of sim⁡(I,T)/τ\operatorname{sim}(I,T)/\tau; as in ALBEF, a momentum encoder provides soft targets yy instead of pure one-hot labels
pitmp^{itm}
2-way softmax from the [Encode] output; negatives are hard negatives sampled from the contrastive similarities
LMLM
autoregressive cross-entropy on the caption, with label smoothing 0.1
CaptionContrastive simsim / τSampling probability
a dog catching a ball on grass0.628.860.725
a puppy running in a park0.557.860.267
two cats on a sofa0.304.290.008
a bowl of noodles0.050.710.0002
Worked example: picking a hard negative for ITM. One image, four in-batch captions that are not its own, τ = 0.07 (computed in Python)

A uniformly sampled negative would be "a bowl of noodles" a quarter of the time, and the ITM head would learn nothing from it. With similarity-weighted sampling, 99% of the negatives are the two dog captions. They differ from the true caption in details, and these are exactly the cases a retrieval re-ranker has to get right.

CapFilt: captioning and filtering the web data

  1. 1Pre-train MED on the noisy data (web pairs plus human-annotated COCO and Visual Genome pairs).
  2. 2Fine-tune two copies separately on COCO: the decoder as a captioner, the grounded encoder as a filter (ITM).
  3. 3For each web image, the captioner samples one synthetic caption.
  4. 4The filter scores both the original web text and the synthetic caption, and drops any text it predicts as unmatched.
  5. 5Pre-train a new model on the kept web texts, the kept synthetic captions and the human-annotated pairs.

Both parts help, and their gains add up. Captions were sampled with nucleus sampling rather than beam search. Nucleus sampling gives more noise, but more diverse captions, and it led to better downstream results. Beam search tends to repeat safe, generic captions.

60 web images · top bar = web alt-text, bottom = synthetic caption · dashed = filtered outtext matches its imagenoisy (does not match)toy rates: web 26/60 correct, captions 49/60filter's ITM score p(match), all 120 texts00.51keep ≥ 0.50kept texts66 (web 22, syn 44)noisy among kept2 → precision 97.0%recall of correct texts85.3%images covered46 / 60raw web only: 60 texts, 34 noisy,precision 43.3%, 26 images coveredBLIP then pre-trains a new model on the kept pairs plus human-annotated ones (COCO, Visual Genome)
CapFilt
fig.A toy web set of 60 images. Only 26 of the 60 alt-texts match their image, so the raw data has 43.3% precision and 26 images with a correct text. Filter alone (threshold 0.5) keeps 22 texts, all correct, so coverage falls to 22. Captioner alone gives 120 texts at 62.5% precision. Both together (default) keep 66 texts, 22 web and 44 synthetic, at 97.0% precision with 46 of 60 images covered and 85.3% recall. At threshold 0.8 noise is zero but recall falls to 62.7% and coverage to 36. The rates are illustrative, not from the paper.

Setup and results (Li et al., 2022)

Data
14M images (COCO, Visual Genome, Conceptual Captions, Conceptual 12M, SBU), optionally plus 115M LAION images for 129M in total
Models
ViT-B/16 or ViT-L/16 image encoder, BERT-base text Transformer; 224 px in pre-training, 384 px for fine-tuning
Reported gains
over the previous best: +2.7% average recall@1 in image–text retrieval, +2.8% CIDEr in captioning, +1.6% VQA score
Transfer
zero-shot transfer to video–language tasks (text-to-video retrieval, video QA)
python
1import torch
2from PIL import Image
3from transformers import BlipProcessor, BlipForConditionalGeneration, BlipForImageTextRetrieval
4
5img = Image.open("web.jpg").convert("RGB")
6alt_text = "blue toy free shipping on all orders"
7
8cap_proc = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
9captioner = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
10itm_proc = BlipProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
11filt = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
12
13with torch.no_grad():
14 out = captioner.generate(**cap_proc(images=img, return_tensors="pt"),
15 do_sample=True, top_p=0.9, max_new_tokens=30) # nucleus sampling
16 synthetic = cap_proc.decode(out[0], skip_special_tokens=True)
17 for text in [alt_text, synthetic]:
18 x = itm_proc(images=img, text=text, return_tensors="pt")
19 p_match = filt(**x).itm_score.softmax(-1)[0, 1].item() # ITM head: [no match, match]
20 print(f"{p_match:.2f} {'keep' if p_match >= 0.5 else 'drop'} {text}")
CapFilt in miniature with Hugging Face transformers: caption with nucleus sampling, then score both texts with the ITM head (not run here)

BLIP-2: freeze both ends

Li, Li, Savarese and Hoi ("BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", ICML 2023) keep a pre-trained image encoder (CLIP ViT-L/14 or EVA-CLIP ViT-g/14) and a pre-trained LLM (OPT or FlanT5) frozen. Only a small Q-Former between them is trained. It is a BERT-base-initialised Transformer (188M parameters) with 32 learned query vectors of width 768 that cross-attend to the frozen image features. Its output is always 32×76832\times768, whatever the image resolution.

Training has two stages. Stage 1 connects the Q-Former to the frozen image encoder with BLIP's three objectives (ITC, ITM and image-grounded text generation), using a different self-attention mask between queries and text for each. Stage 2 projects the 32 query outputs with a linear layer into the LLM's input space as soft visual prompts, and trains with the LLM's language-modelling loss. BLIP-2 beat Flamingo-80B by 8.7% on zero-shot VQAv2 with 54× fewer trainable parameters (see Multimodal transformers and Vision-language models).

Check yourself

Which parameters are not shared between BLIP's image-grounded text encoder and its text decoder?

Check yourself

Why does BLIP sample synthetic captions with nucleus sampling instead of beam search?

Remember

  • ✓BLIP (Li, Li, Xiong, Hoi, ICML 2022): one ViT plus a BERT-base text Transformer used as unimodal encoder (ITC), image-grounded encoder (ITM) and image-grounded decoder (LM).
  • ✓Encoder and decoder share everything except self-attention; cross-attention to image patches is inserted between self-attention and feed-forward.
  • ✓ITM trains on hard negatives sampled by contrastive similarity; ITC uses momentum-distilled soft targets as in ALBEF.
  • ✓CapFilt: a COCO-fine-tuned captioner (nucleus sampling) adds synthetic captions, a COCO-fine-tuned ITM filter drops mismatched web and synthetic texts, and a new model is pre-trained on the result.
  • ✓BLIP-2: frozen image encoder + frozen LLM bridged by a Q-Former with 32 learned queries (32 × 768 output), trained in two stages.
← back to Deep Learning roadmap