SigLIP

SigLIP replaces CLIP's batch-wide softmax with an independent sigmoid on every image–text pair, plus a learned bias (init t = 10, b = −10). No global normalisation means a cheap chunked loss and better results at small batches.

7 min read661 wordsadvanced0 viewsupdated 2026-10-15

Definition

SigLIP (Sigmoid loss for Language–Image Pre-training) trains a CLIP-style dual encoder with a pairwise binary objective: each of the ∣B∣2|B|^2 image–text pairs in a batch is classified as match or non-match by a sigmoid, independently of the others. Introduced by Zhai, Mustafa, Kolesnikov and Beyer, "Sigmoid Loss for Language Image Pre-Training" (Google DeepMind, ICCV 2023).

The task is the same as in CLIP: embed images and texts so that matching pairs have high cosine similarity, then use the text side for zero-shot classification and retrieval. Only the loss changes. CLIP's symmetric InfoNCE loss is a softmax over each row and each column of the ∣B∣×∣B∣|B|\times|B| logit matrix. Every probability depends on the whole batch. That has two costs.

First, computation: the softmax denominator of each row needs every text in the batch, so distributed training all-gathers all embeddings and materialises the full matrix. Second, behaviour: the loss is relative. A positive pair is only "right" in comparison with the other captions in the batch, so small batches give a weak signal, and two near-duplicate pairs compete for the same probability mass.

The sigmoid loss

L=−1∣B∣∑i=1∣B∣∑j=1∣B∣log⁡σ ⁣(zij (t xi ⁣⋅yj+b)),zij={+1i=j−1i≠j,t=et′\mathcal L=-\frac{1}{|B|}\sum_{i=1}^{|B|}\sum_{j=1}^{|B|}\log\sigma\!\big(z_{ij}\,(t\,x_i\!\cdot y_j+b)\big),\qquad z_{ij}=\begin{cases}+1 & i=j\\-1 & i\ne j\end{cases},\qquad t=e^{t'}
xi, yjx_i,\,y_j
L2-normalised image and text embeddings
t′t'
learnable log-temperature, initialised to log⁡10\log 10, so t=10t=10 at the start
bb
learnable bias, initialised to −10-10; the decision boundary σ=0.5\sigma=0.5 sits at xi⋅yj=−b/tx_i\cdot y_j=-b/t
1∣B∣\tfrac{1}{|B|}
a mean over images and a sum over texts, so the ∣B∣−1|B|-1 negatives per image are all counted

Compare with the CLIP row term −log⁡(et xi⋅yi/∑jet xi⋅yj)-\log\big(e^{t\,x_i\cdot y_i}/\sum_j e^{t\,x_i\cdot y_j}\big). A bias added to every logit cancels inside a softmax, so CLIP has no use for bb. In SigLIP it matters. Each image has 1 positive and ∣B∣−1|B|-1 negatives. With b=0b=0 and random embeddings, every pair starts near σ=0.5\sigma=0.5 and the loss is about ∣B∣log⁡2|B|\log2 per image, about 22,713 for ∣B∣=32,768|B|=32{,}768, almost all of it from negatives. With b=−10b=-10 every pair starts predicted "no match": the positive term is log⁡(1+e10)=10.00\log(1+e^{10})=10.00, and the 32,767 negatives together add only 32,767⋅log⁡(1+e−10)=1.4932{,}767\cdot\log(1+e^{-10})=1.49. Training starts close to the prior instead of spending its first steps correcting a huge imbalance.

Chunked implementation

Because the loss is a plain sum over pairs, it can be computed block by block. Each of DD devices holds b=∣B∣/Db=|B|/D pairs. It first scores its own images against its own texts (this block holds the positives). The text embeddings are then passed to the neighbouring device, and each device scores a block of pure negatives. After DD steps every block has been covered.

chunked sigmoid loss: |B| = 12 pairs on 3 devices, b = 4 pairs eachstep 1: own textspositives + negativesstep 2: texts permutedone device on, negativesstep 3: permuted againnegatives; donerows: images · columns: textspositive pair (zᵢᵢ = +1)negative (zᵢⱼ = −1)device 14 images, 4 textsdevice 24 images, 4 textsdevice 34 images, 4 textstext embeddings move one device per step (collective permute)logits held per device: b² = 16,not |B|² = 144loss = sum of the block losses(3 steps = number of devices)softmax (CLIP): each row needs Σⱼ exp over all |B| texts, so embeddings are all-gathered firstsigmoid (SigLIP): each cell is a binary term −log σ(zᵢⱼ(t·xᵢ·yⱼ + b)), so blocks add up exactly
fig.12 pairs on 3 devices. Step 1 computes the three diagonal 4×4 blocks, which contain all 12 positives. Steps 2 and 3 pass the text embeddings one device on and compute blocks of negatives only. Each device holds a 4×4 block of logits (16) at a time instead of the full 12×12 (144). With a softmax this is impossible without first gathering every text, because each row's denominator spans all 12 columns.

Memory per device for the logits is b2b^2 instead of ∣B∣2|B|^2, and there is no all-gather. That is what let the authors push batch size to one million in their study.

Findings and models (Zhai et al., 2023)

Small batches
the sigmoid loss beats the softmax loss clearly at small batch sizes; the gap closes as batches grow
Batch size
tested up to 1M; gains saturate, and about 32k is enough for both losses
SigLiT
with a frozen pre-trained image tower (LiT), 84.5% zero-shot ImageNet after two days on four TPUv4 chips
mSigLIP
a multilingual variant trained on WebLI with captions in over 100 languages
Released towers
SigLIP So400m/14 (a shape-optimised ~400M-parameter ViT) is the image encoder of PaliGemma; SigLIP 2 (Tschannen et al., 2025) adds further training objectives on top of the sigmoid loss
batch N = 8 · 64 pairs: 8 positives (diagonal), 56 negatives · t = 20, b = -10CLIP: softmax over each row of t·SSigLIP: σ(t·Sᵢⱼ + b), every cell alonetexts 1…8 →images 1…8 →texts 1…8 →images 1…8 →.98000.0200001.00000000001.00000000001.00000000001.00000000001.00000000001.0000000.0100.99.95000.320000.9900000000.9900.0200000.910000.01000.9300000000.990.090.010000.9000000.96001.00L = ½(0.004 + 0.121) = 0.063image→text + text→image CE; b has no effectL = 0.043 + 0.457 = 0.500positives + negatives, summed per imageσ = 0.5 where cos = −b/t = 0.50: pairs above count as matches, independently of the rest of thebatchchance level for the softmax: ln N = 2.079 · mean cos: positives 0.702, negatives -0.050
presets
fig.One batch of N = 8 pairs (alignment 0.60), scored both ways at t = 20, b = −10. The softmax rows give p₁₁ = 0.975 and loss ½(0.004 + 0.121) = 0.063. The sigmoid gives σ₁₁ = 0.949 and loss 0.043 + 0.457 = 0.500. Image 8 is also close to text 5 (cos 0.654): the sigmoid scores that negative 0.956 and charges for it, while its softmax share is only 0.006 because image 8's own caption (cos 0.912) wins the row. Turn on "duplicate pair": softmax p₁₁ falls to 0.306 because the duplicate caption takes the row's mass, while sigmoid σ₁₁ stays 0.949 and the negative term rises from 0.457 to 1.315, of which the two false-negative cells contribute 0.828. Click "no bias": the sigmoid loss jumps to 11.611, all of it from the 56 negatives. At the init preset (t = 10, b = −10) the sigmoid loss is 3.066, dominated by positives that have not yet crossed −b/t = 1.

Worked example: both losses on one 3×3 batch

Cosines S=(0.80.20.10.30.70.00.10.40.6)S=\begin{pmatrix}0.8&0.2&0.1\\0.3&0.7&0.0\\0.1&0.4&0.6\end{pmatrix}, t=10t=10, b=−5b=-5.

QuantityRow 1Row 2Row 3Loss
sigmoid logits t·S + b3, −3, −4−2, 2, −5−4, −1, 1
σ of the logits0.953, 0.047, 0.0180.119, 0.881, 0.0070.018, 0.269, 0.731
−log σ(z·logit), per cell0.049, 0.049, 0.0180.127, 0.127, 0.0070.018, 0.313, 0.313sum / 3 = 0.340
softmax row p(i→t) of t·S0.997, 0.002, 0.0010.018, 0.981, 0.0010.006, 0.118, 0.876
softmax CE per row / per column0.003 / 0.0080.019 / 0.0550.133 / 0.009½(0.052 + 0.024) = 0.038
Computed in Python (natural logs)

The sigmoid probabilities in a row need not sum to 1: row 3 has 0.731 for its positive and 0.269 for text 2. The (3, 2) cell costs 0.313 in the sigmoid loss, as much as the weak positive (3, 3). The softmax rescales the same row to 0.876 for the positive, and its loss would not change if all three logits in the row shifted together.

python
1import numpy as np
2
3def sigmoid_loss(x, y, t, b):
4 """x, y: (n, d) unit-norm image / text embeddings. Sum over texts, mean over images."""
5 logits = t * x @ y.T + b
6 z = 2 * np.eye(len(x)) - 1 # +1 on the diagonal, -1 elsewhere
7 return np.logaddexp(0, -z * logits).sum() / len(x) # -log σ(z·logits), computed stably
8
9def chunked_sigmoid_loss(x, y, t, b, devices):
10 xs, ys = np.split(x, devices), np.split(y, devices)
11 total = 0.0
12 for k in range(devices): # step k: texts have moved k devices on
13 for d in range(devices): # (in parallel on real hardware)
14 logits = t * xs[d] @ ys[(d + k) % devices].T + b
15 z = 2 * np.eye(len(logits)) - 1 if k == 0 else -np.ones_like(logits)
16 total += np.logaddexp(0, -z * logits).sum()
17 return total / len(x)
18
19rng = np.random.default_rng(0)
20x = rng.normal(size=(12, 64)); x /= np.linalg.norm(x, axis=1, keepdims=True)
21y = x + 0.5 * rng.normal(size=(12, 64)); y /= np.linalg.norm(y, axis=1, keepdims=True)
22t, b = np.exp(np.log(10.0)), -10.0 # SigLIP initialisation: t' = log 10
23print(round(sigmoid_loss(x, y, t, b), 4), round(chunked_sigmoid_loss(x, y, t, b, devices=3), 4))
The sigmoid loss in numpy, full and chunked over 3 simulated devices; both print 7.0789 (run here)

Use when

  • +No global normalisation: chunked computation, b² logits per device, no all-gather
  • +Stronger than softmax at small and medium batch sizes
  • +Pairs are scored independently, so near-duplicates in a batch do not steal probability from each other
  • +Calibrated per-pair scores allow multi-label answers and thresholds

Avoid when

  • −Two scalars to initialise well; with b = 0 the loss starts dominated by negatives
  • −At very large batches it matches the softmax loss rather than beating it
  • −Scores are not comparable to CLIP probabilities, which trips up code written for CLIP

Check yourself

In a batch of 4 pairs, you add 3 to every logit in image 1's row. How do the two losses respond?

Check yourself

With random embeddings (all cosines ≈ 0) and |B| = 32,768, what is the SigLIP loss per image at initialisation (t = 10, b = −10)?

Remember

  • ✓SigLIP loss: −1∣B∣∑i,jlog⁡σ(zij(t xi⋅yj+b))-\frac1{|B|}\sum_{i,j}\log\sigma(z_{ij}(t\,x_i\cdot y_j+b)) with zij=±1z_{ij}=\pm1; every pair is a separate binary decision.
  • ✓Initialise t′=log⁡10t'=\log10 (t = 10) and b=−10b=-10. The bias offsets the 1:(∣B∣−1)1:(|B|-1) positive–negative imbalance; in a softmax it would cancel.
  • ✓No global normalisation, so the loss is computed in chunks: pass text embeddings between devices, b² logits per device, no all-gather.
  • ✓Beats the softmax loss at small batches; both saturate around 32k. SigLiT reached 84.5% zero-shot ImageNet on four TPUv4 chips in two days.
  • ✓At inference use a sigmoid on the logits (scores do not sum to 1) and pad text to the training length.
← back to Deep Learning roadmap