Definition
SigLIP (Sigmoid loss for Language–Image Pre-training) trains a CLIP-style dual encoder with a pairwise binary objective: each of the image–text pairs in a batch is classified as match or non-match by a sigmoid, independently of the others. Introduced by Zhai, Mustafa, Kolesnikov and Beyer, "Sigmoid Loss for Language Image Pre-Training" (Google DeepMind, ICCV 2023).
The task is the same as in CLIP: embed images and texts so that matching pairs have high cosine similarity, then use the text side for zero-shot classification and retrieval. Only the loss changes. CLIP's symmetric InfoNCE loss is a softmax over each row and each column of the logit matrix. Every probability depends on the whole batch. That has two costs.
First, computation: the softmax denominator of each row needs every text in the batch, so distributed training all-gathers all embeddings and materialises the full matrix. Second, behaviour: the loss is relative. A positive pair is only "right" in comparison with the other captions in the batch, so small batches give a weak signal, and two near-duplicate pairs compete for the same probability mass.
The sigmoid loss
- L2-normalised image and text embeddings
- learnable log-temperature, initialised to , so at the start
- learnable bias, initialised to ; the decision boundary sits at
- a mean over images and a sum over texts, so the negatives per image are all counted
Compare with the CLIP row term . A bias added to every logit cancels inside a softmax, so CLIP has no use for . In SigLIP it matters. Each image has 1 positive and negatives. With and random embeddings, every pair starts near and the loss is about per image, about 22,713 for , almost all of it from negatives. With every pair starts predicted "no match": the positive term is , and the 32,767 negatives together add only . Training starts close to the prior instead of spending its first steps correcting a huge imbalance.
Chunked implementation
Because the loss is a plain sum over pairs, it can be computed block by block. Each of devices holds pairs. It first scores its own images against its own texts (this block holds the positives). The text embeddings are then passed to the neighbouring device, and each device scores a block of pure negatives. After steps every block has been covered.
Memory per device for the logits is instead of , and there is no all-gather. That is what let the authors push batch size to one million in their study.
Findings and models (Zhai et al., 2023)
- Small batches
- the sigmoid loss beats the softmax loss clearly at small batch sizes; the gap closes as batches grow
- Batch size
- tested up to 1M; gains saturate, and about 32k is enough for both losses
- SigLiT
- with a frozen pre-trained image tower (LiT), 84.5% zero-shot ImageNet after two days on four TPUv4 chips
- mSigLIP
- a multilingual variant trained on WebLI with captions in over 100 languages
- Released towers
- SigLIP So400m/14 (a shape-optimised ~400M-parameter ViT) is the image encoder of PaliGemma; SigLIP 2 (Tschannen et al., 2025) adds further training objectives on top of the sigmoid loss
Worked example: both losses on one 3×3 batch
Cosines , , .
| Quantity | Row 1 | Row 2 | Row 3 | Loss |
|---|---|---|---|---|
| sigmoid logits t·S + b | 3, −3, −4 | −2, 2, −5 | −4, −1, 1 | |
| σ of the logits | 0.953, 0.047, 0.018 | 0.119, 0.881, 0.007 | 0.018, 0.269, 0.731 | |
| −log σ(z·logit), per cell | 0.049, 0.049, 0.018 | 0.127, 0.127, 0.007 | 0.018, 0.313, 0.313 | sum / 3 = 0.340 |
| softmax row p(i→t) of t·S | 0.997, 0.002, 0.001 | 0.018, 0.981, 0.001 | 0.006, 0.118, 0.876 | |
| softmax CE per row / per column | 0.003 / 0.008 | 0.019 / 0.055 | 0.133 / 0.009 | ½(0.052 + 0.024) = 0.038 |
The sigmoid probabilities in a row need not sum to 1: row 3 has 0.731 for its positive and 0.269 for text 2. The (3, 2) cell costs 0.313 in the sigmoid loss, as much as the weak positive (3, 3). The softmax rescales the same row to 0.876 for the positive, and its loss would not change if all three logits in the row shifted together.
1import numpy as np23def sigmoid_loss(x, y, t, b):4 """x, y: (n, d) unit-norm image / text embeddings. Sum over texts, mean over images."""5 logits = t * x @ y.T + b6 z = 2 * np.eye(len(x)) - 1 # +1 on the diagonal, -1 elsewhere7 return np.logaddexp(0, -z * logits).sum() / len(x) # -log σ(z·logits), computed stably89def chunked_sigmoid_loss(x, y, t, b, devices):10 xs, ys = np.split(x, devices), np.split(y, devices)11 total = 0.012 for k in range(devices): # step k: texts have moved k devices on13 for d in range(devices): # (in parallel on real hardware)14 logits = t * xs[d] @ ys[(d + k) % devices].T + b15 z = 2 * np.eye(len(logits)) - 1 if k == 0 else -np.ones_like(logits)16 total += np.logaddexp(0, -z * logits).sum()17 return total / len(x)1819rng = np.random.default_rng(0)20x = rng.normal(size=(12, 64)); x /= np.linalg.norm(x, axis=1, keepdims=True)21y = x + 0.5 * rng.normal(size=(12, 64)); y /= np.linalg.norm(y, axis=1, keepdims=True)22t, b = np.exp(np.log(10.0)), -10.0 # SigLIP initialisation: t' = log 1023print(round(sigmoid_loss(x, y, t, b), 4), round(chunked_sigmoid_loss(x, y, t, b, devices=3), 4))Use when
- +No global normalisation: chunked computation, b² logits per device, no all-gather
- +Stronger than softmax at small and medium batch sizes
- +Pairs are scored independently, so near-duplicates in a batch do not steal probability from each other
- +Calibrated per-pair scores allow multi-label answers and thresholds
Avoid when
- −Two scalars to initialise well; with b = 0 the loss starts dominated by negatives
- −At very large batches it matches the softmax loss rather than beating it
- −Scores are not comparable to CLIP probabilities, which trips up code written for CLIP
Check yourself
In a batch of 4 pairs, you add 3 to every logit in image 1's row. How do the two losses respond?
Check yourself
With random embeddings (all cosines ≈ 0) and |B| = 32,768, what is the SigLIP loss per image at initialisation (t = 10, b = −10)?
Remember
- ✓SigLIP loss: with ; every pair is a separate binary decision.
- ✓Initialise (t = 10) and . The bias offsets the positive–negative imbalance; in a softmax it would cancel.
- ✓No global normalisation, so the loss is computed in chunks: pass text embeddings between devices, b² logits per device, no all-gather.
- ✓Beats the softmax loss at small batches; both saturate around 32k. SigLiT reached 84.5% zero-shot ImageNet on four TPUv4 chips in two days.
- ✓At inference use a sigmoid on the logits (scores do not sum to 1) and pad text to the training length.