Definition
BLIP (Bootstrapping Language-Image Pre-training) is a vision–language model that is pre-trained with three objectives on shared weights, so one model can retrieve, classify image–text pairs and write captions. It also improves its own training data by generating and filtering captions. Introduced by Junnan Li, Dongxu Li, Caiming Xiong and Steven Hoi (Salesforce Research), "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation", ICML 2022.
Two problems motivated it. First, models were either encoders or encoder–decoders. Dual encoders such as CLIP are good at retrieval but cannot write text. Fused encoders (e.g. ALBEF) are good at matching and VQA but are not trained to generate. Second, web alt-text is noisy. Many captions describe the page, not the picture ("free shipping on all orders"), and scaling up such data gives diminishing returns.
Multimodal mixture of encoder–decoder (MED)
A ViT image encoder (ViT-B/16 or ViT-L/16, initialised from ImageNet) turns the image into a [CLS] embedding and patch features. A BERT-base text Transformer runs in three modes over the same caption:
- 1Unimodal text encoder. Bidirectional self-attention, no access to the image. Its [CLS] embedding is contrasted with the image [CLS] embedding (ITC).
- 2Image-grounded text encoder. A cross-attention layer between self-attention and feed-forward in every block attends to the patch features. An [Encode] token's output feeds a binary head: does this text match this image (ITM)?
- 3Image-grounded text decoder. The same blocks with causal self-attention and a [Decode] start token, trained to generate the caption (LM).
The encoder and decoder share all parameters except the self-attention layers. The authors found that self-attention is where encoding and decoding conflict, and that sharing everything else gives better results than separate models with fewer parameters. Each pair costs one forward pass of the image encoder and three of the text Transformer.
- softmax over the batch of ; as in ALBEF, a momentum encoder provides soft targets instead of pure one-hot labels
- 2-way softmax from the [Encode] output; negatives are hard negatives sampled from the contrastive similarities
- autoregressive cross-entropy on the caption, with label smoothing 0.1
| Caption | Contrastive sim | sim / τ | Sampling probability |
|---|---|---|---|
| a dog catching a ball on grass | 0.62 | 8.86 | 0.725 |
| a puppy running in a park | 0.55 | 7.86 | 0.267 |
| two cats on a sofa | 0.30 | 4.29 | 0.008 |
| a bowl of noodles | 0.05 | 0.71 | 0.0002 |
A uniformly sampled negative would be "a bowl of noodles" a quarter of the time, and the ITM head would learn nothing from it. With similarity-weighted sampling, 99% of the negatives are the two dog captions. They differ from the true caption in details, and these are exactly the cases a retrieval re-ranker has to get right.
CapFilt: captioning and filtering the web data
- 1Pre-train MED on the noisy data (web pairs plus human-annotated COCO and Visual Genome pairs).
- 2Fine-tune two copies separately on COCO: the decoder as a captioner, the grounded encoder as a filter (ITM).
- 3For each web image, the captioner samples one synthetic caption.
- 4The filter scores both the original web text and the synthetic caption, and drops any text it predicts as unmatched.
- 5Pre-train a new model on the kept web texts, the kept synthetic captions and the human-annotated pairs.
Both parts help, and their gains add up. Captions were sampled with nucleus sampling rather than beam search. Nucleus sampling gives more noise, but more diverse captions, and it led to better downstream results. Beam search tends to repeat safe, generic captions.
Setup and results (Li et al., 2022)
- Data
- 14M images (COCO, Visual Genome, Conceptual Captions, Conceptual 12M, SBU), optionally plus 115M LAION images for 129M in total
- Models
- ViT-B/16 or ViT-L/16 image encoder, BERT-base text Transformer; 224 px in pre-training, 384 px for fine-tuning
- Reported gains
- over the previous best: +2.7% average recall@1 in image–text retrieval, +2.8% CIDEr in captioning, +1.6% VQA score
- Transfer
- zero-shot transfer to video–language tasks (text-to-video retrieval, video QA)
1import torch2from PIL import Image3from transformers import BlipProcessor, BlipForConditionalGeneration, BlipForImageTextRetrieval45img = Image.open("web.jpg").convert("RGB")6alt_text = "blue toy free shipping on all orders"78cap_proc = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")9captioner = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")10itm_proc = BlipProcessor.from_pretrained("Salesforce/blip-itm-base-coco")11filt = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")1213with torch.no_grad():14 out = captioner.generate(**cap_proc(images=img, return_tensors="pt"),15 do_sample=True, top_p=0.9, max_new_tokens=30) # nucleus sampling16 synthetic = cap_proc.decode(out[0], skip_special_tokens=True)17 for text in [alt_text, synthetic]:18 x = itm_proc(images=img, text=text, return_tensors="pt")19 p_match = filt(**x).itm_score.softmax(-1)[0, 1].item() # ITM head: [no match, match]20 print(f"{p_match:.2f} {'keep' if p_match >= 0.5 else 'drop'} {text}")BLIP-2: freeze both ends
Li, Li, Savarese and Hoi ("BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", ICML 2023) keep a pre-trained image encoder (CLIP ViT-L/14 or EVA-CLIP ViT-g/14) and a pre-trained LLM (OPT or FlanT5) frozen. Only a small Q-Former between them is trained. It is a BERT-base-initialised Transformer (188M parameters) with 32 learned query vectors of width 768 that cross-attend to the frozen image features. Its output is always , whatever the image resolution.
Training has two stages. Stage 1 connects the Q-Former to the frozen image encoder with BLIP's three objectives (ITC, ITM and image-grounded text generation), using a different self-attention mask between queries and text for each. Stage 2 projects the 32 query outputs with a linear layer into the LLM's input space as soft visual prompts, and trains with the LLM's language-modelling loss. BLIP-2 beat Flamingo-80B by 8.7% on zero-shot VQAv2 with 54× fewer trainable parameters (see Multimodal transformers and Vision-language models).
Check yourself
Which parameters are not shared between BLIP's image-grounded text encoder and its text decoder?
Check yourself
Why does BLIP sample synthetic captions with nucleus sampling instead of beam search?
Remember
- ✓BLIP (Li, Li, Xiong, Hoi, ICML 2022): one ViT plus a BERT-base text Transformer used as unimodal encoder (ITC), image-grounded encoder (ITM) and image-grounded decoder (LM).
- ✓Encoder and decoder share everything except self-attention; cross-attention to image patches is inserted between self-attention and feed-forward.
- ✓ITM trains on hard negatives sampled by contrastive similarity; ITC uses momentum-distilled soft targets as in ALBEF.
- ✓CapFilt: a COCO-fine-tuned captioner (nucleus sampling) adds synthetic captions, a COCO-fine-tuned ITM filter drops mismatched web and synthetic texts, and a new model is pre-trained on the result.
- ✓BLIP-2: frozen image encoder + frozen LLM bridged by a Q-Former with 32 learned queries (32 × 768 output), trained in two stages.