# SigLIP

> SigLIP replaces CLIP's batch-wide softmax with an independent sigmoid on every image–text pair, plus a learned bias (init t = 10, b = −10). No global normalisation means a cheap chunked loss and better results at small batches.

Source: https://www.mlfoundry.in/learn/siglip  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: advanced · 7 min read · updated 2026-10-15

**SigLIP:** (Sigmoid loss for Language–Image Pre-training) trains a CLIP-style dual encoder with a **pairwise binary** objective: each of the $|B|^2$ image–text pairs in a batch is classified as match or non-match by a sigmoid, independently of the others. Introduced by Zhai, Mustafa, Kolesnikov and Beyer, "Sigmoid Loss for Language Image Pre-Training" (Google DeepMind, ICCV 2023).

The task is the same as in **CLIP**: embed images and texts so that matching pairs have high cosine similarity, then use the text side for zero-shot classification and retrieval. Only the loss changes. CLIP's symmetric **InfoNCE loss** is a softmax over each row and each column of the $|B|\times|B|$ logit matrix. Every probability depends on the whole batch. That has two costs.

First, computation: the softmax denominator of each row needs every text in the batch, so distributed training all-gathers all embeddings and materialises the full matrix. Second, behaviour: the loss is relative. A positive pair is only "right" in comparison with the other captions in the batch, so small batches give a weak signal, and two near-duplicate pairs compete for the same probability mass.

## The sigmoid loss

$$
\mathcal L=-\frac{1}{|B|}\sum_{i=1}^{|B|}\sum_{j=1}^{|B|}\log\sigma\!\big(z_{ij}\,(t\,x_i\!\cdot y_j+b)\big),\qquad z_{ij}=\begin{cases}+1 & i=j\\-1 & i\ne j\end{cases},\qquad t=e^{t'}
$$

- $x_i,\,y_j$: L2-normalised image and text embeddings
- $t'$: learnable log-temperature, initialised to $\log 10$, so $t=10$ at the start
- $b$: learnable bias, initialised to $-10$; the decision boundary $\sigma=0.5$ sits at $x_i\cdot y_j=-b/t$
- $\tfrac{1}{|B|}$: a mean over images and a **sum** over texts, so the $|B|-1$ negatives per image are all counted

Compare with the CLIP row term $-\log\big(e^{t\,x_i\cdot y_i}/\sum_j e^{t\,x_i\cdot y_j}\big)$. A bias added to every logit cancels inside a softmax, so CLIP has no use for $b$. In SigLIP it matters. Each image has 1 positive and $|B|-1$ negatives. With $b=0$ and random embeddings, every pair starts near $\sigma=0.5$ and the loss is about $|B|\log2$ per image, about 22,713 for $|B|=32{,}768$, almost all of it from negatives. With $b=-10$ every pair starts predicted "no match": the positive term is $\log(1+e^{10})=10.00$, and the 32,767 negatives together add only $32{,}767\cdot\log(1+e^{-10})=1.49$. Training starts close to the prior instead of spending its first steps correcting a huge imbalance.

## Chunked implementation

Because the loss is a plain sum over pairs, it can be computed block by block. Each of $D$ devices holds $b=|B|/D$ pairs. It first scores its own images against its own texts (this block holds the positives). The text embeddings are then passed to the neighbouring device, and each device scores a block of pure negatives. After $D$ steps every block has been covered.

> Figure: 12 pairs on 3 devices. Step 1 computes the three diagonal 4×4 blocks, which contain all 12 positives. Steps 2 and 3 pass the text embeddings one device on and compute blocks of negatives only. Each device holds a 4×4 block of logits (16) at a time instead of the full 12×12 (144). With a softmax this is impossible without first gathering every text, because each row's denominator spans all 12 columns.

Memory per device for the logits is $b^2$ instead of $|B|^2$, and there is no all-gather. That is what let the authors push batch size to one million in their study.

**Findings and models (Zhai et al., 2023)**

- **Small batches:** the sigmoid loss beats the softmax loss clearly at small batch sizes; the gap closes as batches grow
- **Batch size:** tested up to 1M; gains saturate, and about 32k is enough for both losses
- **SigLiT:** with a frozen pre-trained image tower (LiT), 84.5% zero-shot ImageNet after two days on four TPUv4 chips
- **mSigLIP:** a multilingual variant trained on WebLI with captions in over 100 languages
- **Released towers:** SigLIP So400m/14 (a shape-optimised ~400M-parameter ViT) is the image encoder of PaliGemma; SigLIP 2 (Tschannen et al., 2025) adds further training objectives on top of the sigmoid loss

> Figure: One batch of N = 8 pairs (alignment 0.60), scored both ways at t = 20, b = −10. The softmax rows give p₁₁ = 0.975 and loss ½(0.004 + 0.121) = 0.063. The sigmoid gives σ₁₁ = 0.949 and loss 0.043 + 0.457 = 0.500. Image 8 is also close to text 5 (cos 0.654): the sigmoid scores that negative 0.956 and charges for it, while its softmax share is only 0.006 because image 8's own caption (cos 0.912) wins the row. Turn on "duplicate pair": softmax p₁₁ falls to 0.306 because the duplicate caption takes the row's mass, while sigmoid σ₁₁ stays 0.949 and the negative term rises from 0.457 to 1.315, of which the two false-negative cells contribute 0.828. Click "no bias": the sigmoid loss jumps to 11.611, all of it from the 56 negatives. At the init preset (t = 10, b = −10) the sigmoid loss is 3.066, dominated by positives that have not yet crossed −b/t = 1.

## Worked example: both losses on one 3×3 batch

Cosines $S=\begin{pmatrix}0.8&0.2&0.1\\0.3&0.7&0.0\\0.1&0.4&0.6\end{pmatrix}$, $t=10$, $b=-5$.

Computed in Python (natural logs)

| Quantity | Row 1 | Row 2 | Row 3 | Loss |
| --- | --- | --- | --- | --- |
| sigmoid logits t·S + b | 3, −3, −4 | −2, 2, −5 | −4, −1, 1 |  |
| σ of the logits | 0.953, 0.047, 0.018 | 0.119, 0.881, 0.007 | 0.018, 0.269, 0.731 |  |
| −log σ(z·logit), per cell | 0.049, 0.049, 0.018 | 0.127, 0.127, 0.007 | 0.018, 0.313, 0.313 | sum / 3 = 0.340 |
| softmax row p(i→t) of t·S | 0.997, 0.002, 0.001 | 0.018, 0.981, 0.001 | 0.006, 0.118, 0.876 |  |
| softmax CE per row / per column | 0.003 / 0.008 | 0.019 / 0.055 | 0.133 / 0.009 | ½(0.052 + 0.024) = 0.038 |

The sigmoid probabilities in a row need not sum to 1: row 3 has 0.731 for its positive and 0.269 for text 2. The (3, 2) cell costs 0.313 in the sigmoid loss, as much as the weak positive (3, 3). The softmax rescales the same row to 0.876 for the positive, and its loss would not change if all three logits in the row shifted together.

```python
import numpy as np

def sigmoid_loss(x, y, t, b):
    """x, y: (n, d) unit-norm image / text embeddings. Sum over texts, mean over images."""
    logits = t * x @ y.T + b
    z = 2 * np.eye(len(x)) - 1                           # +1 on the diagonal, -1 elsewhere
    return np.logaddexp(0, -z * logits).sum() / len(x)   # -log σ(z·logits), computed stably

def chunked_sigmoid_loss(x, y, t, b, devices):
    xs, ys = np.split(x, devices), np.split(y, devices)
    total = 0.0
    for k in range(devices):                             # step k: texts have moved k devices on
        for d in range(devices):                         # (in parallel on real hardware)
            logits = t * xs[d] @ ys[(d + k) % devices].T + b
            z = 2 * np.eye(len(logits)) - 1 if k == 0 else -np.ones_like(logits)
            total += np.logaddexp(0, -z * logits).sum()
    return total / len(x)

rng = np.random.default_rng(0)
x = rng.normal(size=(12, 64)); x /= np.linalg.norm(x, axis=1, keepdims=True)
y = x + 0.5 * rng.normal(size=(12, 64)); y /= np.linalg.norm(y, axis=1, keepdims=True)
t, b = np.exp(np.log(10.0)), -10.0                      # SigLIP initialisation: t' = log 10
print(round(sigmoid_loss(x, y, t, b), 4), round(chunked_sigmoid_loss(x, y, t, b, devices=3), 4))
```

The sigmoid loss in numpy, full and chunked over 3 simulated devices; both print 7.0789 (run here)

> **Read SigLIP outputs with a sigmoid, and pad like training:** SigLIP scores are independent probabilities. Apply `torch.sigmoid` to `logits_per_image`, not a softmax: the scores of different labels need not sum to 1, and "none of these" is a valid answer. The Hugging Face checkpoints were trained with text padded to a fixed length (64 tokens), so tokenise with `padding="max_length"`. Dynamic padding changes the scores, usually for the worse.

> **Why the bias matters more than the loss form:** The positives-to-negatives ratio is $1:(|B|-1)$. A softmax is immune to this because the normalisation makes it a 1-of-$|B|$ choice. A sum of binary terms is not: without a prior the negatives dominate. $b$ is that prior. During training $t$ and $b$ both move, and the boundary $-b/t$ settles where matched and unmatched cosines separate.

**Strengths**

- No global normalisation: chunked computation, b² logits per device, no all-gather
- Stronger than softmax at small and medium batch sizes
- Pairs are scored independently, so near-duplicates in a batch do not steal probability from each other
- Calibrated per-pair scores allow multi-label answers and thresholds

**Limitations**

- Two scalars to initialise well; with b = 0 the loss starts dominated by negatives
- At very large batches it matches the softmax loss rather than beating it
- Scores are not comparable to CLIP probabilities, which trips up code written for CLIP

**Question:** In a batch of 4 pairs, you add 3 to every logit in image 1's row. How do the two losses respond?

- A. Both are unchanged
- B. The softmax row term is unchanged; the sigmoid loss changes, because each cell is scored on its own
- C. The softmax row term changes; the sigmoid loss is unchanged
- D. Both change by the same amount

**Answer:** B. The softmax row term is unchanged; the sigmoid loss changes, because each cell is scored on its own — A constant shift cancels between the numerator and denominator of a softmax, so CLIP's image→text term for that row is unchanged. In SigLIP the positive gets more likely and the three negatives get less likely to be rejected, so the loss changes. That is also why CLIP needs no bias $b$ and SigLIP does.

**Question:** With random embeddings (all cosines ≈ 0) and |B| = 32,768, what is the SigLIP loss per image at initialisation (t = 10, b = −10)?

- A. ≈ 0.69
- B. ≈ 10.4 (= ln 32,768)
- C. ≈ 11.5
- D. ≈ 22,713

**Answer:** C. ≈ 11.5 — All logits equal $b=-10$. The positive costs $\log(1+e^{10})=10.00$, and the 32,767 negatives cost $\log(1+e^{-10})=4.5\times10^{-5}$ each, 1.49 in total, so about 11.5. With $b=0$ it would be $32{,}768\cdot\ln2\approx22{,}713$.

**Key points**

- SigLIP loss: $-\frac1{|B|}\sum_{i,j}\log\sigma(z_{ij}(t\,x_i\cdot y_j+b))$ with $z_{ij}=\pm1$; every pair is a separate binary decision.
- Initialise $t'=\log10$ (t = 10) and $b=-10$. The bias offsets the $1:(|B|-1)$ positive–negative imbalance; in a softmax it would cancel.
- No global normalisation, so the loss is computed in chunks: pass text embeddings between devices, b² logits per device, no all-gather.
- Beats the softmax loss at small batches; both saturate around 32k. SigLiT reached 84.5% zero-shot ImageNet on four TPUv4 chips in two days.
- At inference use a sigmoid on the logits (scores do not sum to 1) and pad text to the training length.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
