# Binary cross-entropy

> The Bernoulli negative log-likelihood of a sigmoid output. On the logit it is softplus(z) − yz, with gradient σ(z) − y. Compute it from logits, use pos_weight for imbalance, and apply one per label for multi-label tasks.

Source: https://www.mlfoundry.in/learn/binary-cross-entropy  
Section: Deep Learning › Loss Functions  
Level: core · 11 min read · updated 2026-10-11

**Binary cross-entropy:** (BCE, log loss) scores a predicted probability $p=\sigma(z)$ of the positive class against a target $y\in\{0,1\}$ (or a soft target $y\in[0,1]$) as $\ell=-[y\ln p+(1-y)\ln(1-p)]$. It is the negative log-likelihood of $y$ under a Bernoulli distribution with parameter $p$.

BCE is the loss for any yes/no output: spam or not, click or not, real or fake in a GAN discriminator, and each label of a multi-label tagger. The **Loss functions** overview showed that it comes from a Bernoulli likelihood and that its logit gradient is $p-y$. This article covers how it is actually used: the logit form that frameworks compute, why that form is needed numerically, how `pos_weight` handles class imbalance, and how one BCE per label gives multi-label classification.

The loss is older than neural networks. It is the log-likelihood of **logistic regression** (Cox, 1958), whose log-odds link Berkson had called the *logit* in 1944. The name "cross-entropy" comes from information theory: for a target distribution $(y,1-y)$ and a model $(p,1-p)$ it is exactly the cross-entropy $H(y,p)$. In 1988 Baum and Wilczek, and independently Solla, Levin and Fleisher, proposed it for training neural networks with probabilistic outputs in place of squared error.

## Formula on the logit

Networks output a logit $z$ (an unbounded real number), not a probability. Substituting $p=\sigma(z)$ and using $\ln\sigma(z)=-\operatorname{softplus}(-z)$ and $\ln(1-\sigma(z))=-\operatorname{softplus}(z)$ turns the loss into a function of $z$ that never takes the log of a rounded probability:

$$
\ell(z,y)=y\,\operatorname{softplus}(-z)+(1-y)\operatorname{softplus}(z)=\operatorname{softplus}(z)-yz=\max(z,0)-yz+\ln\!\big(1+e^{-|z|}\big)
$$

- $z$: the logit, $\ln\frac{p}{1-p}$; the raw output of the last linear layer
- $y$: target: 0 or 1, or any value in $[0,1]$ for a soft label
- $\operatorname{softplus}(x)$: $\ln(1+e^x)$; the last form is the overflow-free way to evaluate it

$$
\frac{\partial\ell}{\partial z}=\sigma(z)-y\qquad\qquad\frac{\partial^2\ell}{\partial z^2}=\sigma(z)\big(1-\sigma(z)\big)\in(0,\tfrac14]
$$

- $\sigma(z)-y$: prediction minus target: bounded in $(-1,1)$, and $\approx\mp1$ when the model is confidently wrong
- $\sigma(1-\sigma)$: positive curvature, so BCE is convex in $z$ (and in the weights of a logistic regression)

For a soft target the minimum is at $\sigma(z^*)=y$, i.e. $z^*=\operatorname{logit}(y)$. A hard target $y=1$ has its minimum at $z\to+\infty$: the loss keeps decreasing as the logit grows, which is why unregularized classifiers keep growing their weights on separable data.

**At a glance**

- **Task:** binary classification; multi-label classification (one BCE per label)
- **Output layer:** one logit per binary decision, **no** sigmoid inside the model when training
- **Range:** $[0,\infty)$; $\ln 2=0.693$ at $p=0.5$
- **Gradient w.r.t. z:** $\sigma(z)-y$; with `pos_weight` $w$: $\sigma(z)(wy+1-y)-wy$
- **Probabilistic model:** $y\sim\mathrm{Bernoulli}(\sigma(z))$
- **PyTorch:** `nn.BCEWithLogitsLoss`, `F.binary_cross_entropy_with_logits`; `nn.BCELoss` only for probabilities
- **Keras:** `keras.losses.BinaryCrossentropy(from_logits=True)`

> Figure: Top: the loss as a function of the logit for the chosen target (solid), with the weighted y = 1 branch and the y = 0 branch dashed. Bottom: the gradient. At the default z = −2, y = 1 the loss is 2.1269 and the gradient −0.8808: a wrong, confident logit gets almost the full unit push. Slide y to a soft value and the minimum z* moves to logit(y); at y = 0.2 it sits at −1.386. Raise pos_weight and the y = 1 branch is scaled by w, moving z* by ln w: the '20% positives, w = 4' preset puts z* at exactly 0. 'Overflow test' sets z = 20, y = 0: the logit form returns 20.0000, but sigmoid rounded to float32 is exactly 1 and BCELoss reports its clamp value 100.

## Why the logit form is not optional

The two-step recipe `p = torch.sigmoid(z)` then `nn.BCELoss()(p, y)` is mathematically identical and numerically broken. In float32, $\sigma(z)$ rounds to exactly 1 once $e^{-z}$ is below half a unit in the last place of 1, which happens near $z\approx16.6$. From there on, a negative example has $1-p=0$ and $\ln 0=-\infty$. PyTorch's `BCELoss` clamps each log term at $-100$ to avoid infinities, so the loss reads 100 instead of the true value, and the gradient computed from the rounded probability no longer resembles the true $\sigma(z)-y$.

Negative example (y = 0), loss in nats: exact logit form vs sigmoid rounded to float32 followed by BCELoss (computed by the figure's float32 emulation)

| logit z | exact softplus(z) | σ → BCELoss (float32) | σ(z) in float32 |
| --- | --- | --- | --- |
| 10 | 10.0000 | 9.9996 | 0.99995458 |
| 15 | 15.0000 | 14.8438 | 0.99999964 |
| 16 | 16.0000 | 15.9424 | 0.99999988 |
| 17 | 17.0000 | 100 (clamped) | exactly 1 |
| 30 | 30.0000 | 100 (clamped) | exactly 1 |

The positive side is safe in the naive version only because float32 can represent tiny probabilities ($\sigma(-30)\approx9.4\times10^{-14}$) but cannot represent numbers just below 1. The logit form avoids the asymmetry: $\max(z,0)-yz+\ln(1+e^{-|z|})$ only ever exponentiates a non-positive number. It also lets autograd differentiate the stable expression, giving the clean $\sigma(z)-y$ instead of a ratio of rounded quantities.

> **Sigmoid twice:** If the model ends in `nn.Sigmoid()` and you switch to `BCEWithLogitsLoss`, the loss applies a second sigmoid. Every "logit" is then in $(0,1)$, so the predicted probability is stuck in $(0.5,0.73)$, the loss cannot drop below $-\ln\sigma(1)=0.313$ for positives, and negatives can never fall below $p=0.5$. Training runs without errors and plateaus. Keep the model's output raw and apply `torch.sigmoid` only at inference.

## Class imbalance: pos_weight

When positives are rare, the average loss is dominated by easy negatives and a model can score well by predicting "no" everywhere. `pos_weight` multiplies the positive term only:

$$
\ell_w(z,y)=-\big[w\,y\ln\sigma(z)+(1-y)\ln(1-\sigma(z))\big],\qquad z^*=\ln\frac{w\,\pi}{1-\pi}
$$

- $w$: `pos_weight`; $w>1$ raises recall, $w<1$ raises precision
- $\pi$: fraction of positives among examples that look alike (the soft target the model sees on average)
- $z^*$: best constant logit: $\operatorname{logit}(\pi)+\ln w$

The minimizer shows exactly what the weight does: it **shifts the optimal logit by** $\ln w$. With 20% positives the unweighted optimum is $\sigma(z^*)=0.2$, $z^*=-1.386$. PyTorch's documentation suggests $w=\#\text{neg}/\#\text{pos}$, here $0.8/0.2=4$, which moves $z^*$ to $\ln(4\cdot0.2/0.8)=0$, so the model outputs 0.5 where positives are 20%. That is the point of weighting, and also its cost: the outputs are no longer calibrated probabilities. To recover $P(y=1\mid x)$ subtract $\ln w$ from the logit before the sigmoid.

> **weight is not pos_weight:** `BCEWithLogitsLoss(weight=…)` rescales the loss of each element (for example per-sample or per-label importance) and does not distinguish positives from negatives. `pos_weight` must broadcast against the label dimension: for $L$ labels pass a tensor of shape $(L,)$, one ratio per label.

## Multi-label: one BCE per label

In multi-label classification each example can have any subset of $L$ labels (a photo can contain both a dog and a beach). The model outputs $L$ logits, each label gets its own sigmoid, and the loss is the mean of $L$ independent BCE terms. Unlike softmax, the probabilities do not compete and need not sum to 1.

$$
\mathcal L(x)=\frac1L\sum_{j=1}^{L}\Big[\operatorname{softplus}(z_j)-y_jz_j\Big],\qquad y\in\{0,1\}^L
$$

- $z_j$: logit of label $j$
- $y$: multi-hot target vector (float), e.g. $(1,0,1)$

Example: logits $(1.2,\,-0.4,\,2.5)$ with target $(1,0,1)$ give $p=(0.7685,\,0.4013,\,0.9241)$, which sum to 2.094, and per-label losses $(0.2633,\,0.5130,\,0.0789)$ with mean 0.2851. Each label has its own decision threshold, tuned on validation data.

> **Two classes: BCE or softmax?:** For a single binary decision, one logit with BCE and two logits with softmax cross-entropy are the same model: softmax over $(z_1,z_0)$ equals $\sigma(z_1-z_0)$. One logit is simpler and has no redundant parameter. Softmax is right when classes are mutually exclusive and $K>2$; BCE is right when labels can co-occur.

## Worked example

A batch of four examples, mean reduction (checked numerically in Node). Rightmost columns use pos_weight w = 3.

| logit z | y | p = σ(z) | loss | ∂ℓ/∂z = p − y | loss, w = 3 | ∂ℓ/∂z, w = 3 |
| --- | --- | --- | --- | --- | --- | --- |
| 2.0 | 1 | 0.8808 | 0.1269 | −0.1192 | 0.3808 | −0.3576 |
| −1.0 | 1 | 0.2689 | 1.3133 | −0.7311 | 3.9398 | −2.1932 |
| 0.5 | 0 | 0.6225 | 0.9741 | 0.6225 | 0.9741 | 0.6225 |
| −3.0 | 0 | 0.0474 | 0.0486 | 0.0474 | 0.0486 | 0.0474 |

**Reading the table**

1. Example 2 is a missed positive: $\ell=\operatorname{softplus}(-1)=\ln(1+e)=1.3133$ and the gradient $0.2689-1=-0.7311$ pushes the logit up hard.
2. Mean loss $=(0.1269+1.3133+0.9741+0.0486)/4=0.6157$. The per-example gradients of the mean are the column divided by 4.
3. With $w=3$ only the positive rows change: their loss and gradient are multiplied by exactly 3 ($1.3133\to3.9398$, $-0.7311\to-2.1932$). The mean rises to 1.3358, and the negatives' share of the gradient falls.
4. Example 4 is an easy negative. Its gradient 0.0474 is small but never exactly zero, so easy examples still pull a little. Focal loss (Lin et al., 2017) multiplies by $(1-p_t)^\gamma$ to suppress them further.

```python
import torch, torch.nn as nn, torch.nn.functional as F

z = torch.tensor([2.0, -1.0, 0.5, -3.0], requires_grad=True)   # raw logits
y = torch.tensor([1.0, 1.0, 0.0, 0.0])                         # float targets

loss = F.binary_cross_entropy_with_logits(z, y)                # 0.6157
loss.backward()
print(z.grad * 4)                    # sigmoid(z) - y: [-0.1192, -0.7311, 0.6225, 0.0474]

# class imbalance: scale the positive term (suggested: #neg / #pos)
crit = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(3.0))
print(crit(z, y))                    # 1.3358

# the naive two-step version saturates in float32
big = torch.tensor([20.0])
print(nn.BCELoss()(torch.sigmoid(big), torch.tensor([0.0])))           # 100.0 (clamped)
print(F.binary_cross_entropy_with_logits(big, torch.tensor([0.0])))    # 20.0

# multi-label: L logits, multi-hot float targets, one pos_weight per label
logits = torch.tensor([[1.2, -0.4, 2.5]])
target = torch.tensor([[1.0, 0.0, 1.0]])
print(nn.BCEWithLogitsLoss()(logits, target))    # 0.2851
probs = torch.sigmoid(logits)                    # independent: need not sum to 1
pred = probs > 0.5                               # per-label thresholds in practice
```

BCE from logits, pos_weight, and multi-label in PyTorch

**Strengths**

- Proper scoring rule: the expected loss is minimized by the true probability, so outputs are calibrated (without `pos_weight`)
- Gradient $\sigma(z)-y$ is bounded and does not vanish when the model is confidently wrong
- Convex in the logit; logistic regression with BCE has a single optimum
- Handles multi-label tasks and soft targets with no change

**Limitations**

- Unbounded: a single mislabelled example with a confident logit contributes a huge loss
- On separable data the optimum is at $|z|\to\infty$, so it needs weight decay or early stopping
- Easy negatives still dominate the sum under heavy imbalance; `pos_weight` helps but de-calibrates
- The probability form (`BCELoss`) is numerically fragile; only the logit form is safe

**Question:** A detector is trained with BCEWithLogitsLoss and pos_weight = 9 because 10% of examples are positive. On inputs where the true positive rate is 10%, what probability does it output?

- A. 0.1, because the loss is still a proper likelihood
- B. 0.5, because pos_weight shifts the optimal logit by ln 9
- C. 0.9
- D. It depends on the learning rate

**Answer:** B. 0.5, because pos_weight shifts the optimal logit by ln 9 — $z^*=\ln\frac{w\pi}{1-\pi}=\ln\frac{9\cdot0.1}{0.9}=0$, so $\sigma(z^*)=0.5$. The weight moves the decision boundary toward recall; to get calibrated probabilities subtract $\ln 9=2.197$ from the logit before applying the sigmoid.

**Question:** An image can be tagged with any of 20 attributes. Which output head and loss fit?

- A. 20 logits, softmax, cross-entropy
- B. 20 logits, independent sigmoids, BCE averaged over labels
- C. 1 logit, BCE
- D. 20 logits, MSE on the multi-hot vector

**Answer:** B. 20 logits, independent sigmoids, BCE averaged over labels — Softmax forces the 20 probabilities to sum to 1, so tags would compete. Multi-label needs one Bernoulli per attribute: `BCEWithLogitsLoss` on a $(N,20)$ logit tensor with a multi-hot float target.

**Key points**

- BCE is the Bernoulli NLL: $-[y\ln p+(1-y)\ln(1-p)]$, which on the logit is $\operatorname{softplus}(z)-yz$ with gradient $\sigma(z)-y$.
- Always pass logits to `BCEWithLogitsLoss`: in float32, $\sigma(z)$ is exactly 1 above $z\approx16.6$ and `BCELoss` then reports its clamp value 100.
- `pos_weight` $=w$ multiplies the positive term and moves the optimal logit by $\ln w$; it trades calibration for recall.
- Multi-label = one independent sigmoid + BCE per label; probabilities need not sum to 1.
- Soft targets work directly: the minimum is at $\sigma(z)=y$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
