Binary cross-entropy

The Bernoulli negative log-likelihood of a sigmoid output. On the logit it is softplus(z) − yz, with gradient σ(z) − y. Compute it from logits, use pos_weight for imbalance, and apply one per label for multi-label tasks.

11 min read929 wordscore0 viewsupdated 2026-10-11

Definition

Binary cross-entropy (BCE, log loss) scores a predicted probability p=σ(z)p=\sigma(z) of the positive class against a target y∈{0,1}y\in\{0,1\} (or a soft target y∈[0,1]y\in[0,1]) as ℓ=−[yln⁡p+(1−y)ln⁡(1−p)]\ell=-[y\ln p+(1-y)\ln(1-p)]. It is the negative log-likelihood of yy under a Bernoulli distribution with parameter pp.

BCE is the loss for any yes/no output: spam or not, click or not, real or fake in a GAN discriminator, and each label of a multi-label tagger. The Loss functions overview showed that it comes from a Bernoulli likelihood and that its logit gradient is p−yp-y. This article covers how it is actually used: the logit form that frameworks compute, why that form is needed numerically, how pos_weight handles class imbalance, and how one BCE per label gives multi-label classification.

The loss is older than neural networks. It is the log-likelihood of logistic regression (Cox, 1958), whose log-odds link Berkson had called the logit in 1944. The name "cross-entropy" comes from information theory: for a target distribution (y,1−y)(y,1-y) and a model (p,1−p)(p,1-p) it is exactly the cross-entropy H(y,p)H(y,p). In 1988 Baum and Wilczek, and independently Solla, Levin and Fleisher, proposed it for training neural networks with probabilistic outputs in place of squared error.

Formula on the logit

Networks output a logit zz (an unbounded real number), not a probability. Substituting p=σ(z)p=\sigma(z) and using ln⁡σ(z)=−softplus⁡(−z)\ln\sigma(z)=-\operatorname{softplus}(-z) and ln⁡(1−σ(z))=−softplus⁡(z)\ln(1-\sigma(z))=-\operatorname{softplus}(z) turns the loss into a function of zz that never takes the log of a rounded probability:

ℓ(z,y)=y softplus⁡(−z)+(1−y)softplus⁡(z)=softplus⁡(z)−yz=max⁡(z,0)−yz+ln⁡ ⁣(1+e−∣z∣)\ell(z,y)=y\,\operatorname{softplus}(-z)+(1-y)\operatorname{softplus}(z)=\operatorname{softplus}(z)-yz=\max(z,0)-yz+\ln\!\big(1+e^{-|z|}\big)
zz
the logit, ln⁡p1−p\ln\frac{p}{1-p}; the raw output of the last linear layer
yy
target: 0 or 1, or any value in [0,1][0,1] for a soft label
softplus⁡(x)\operatorname{softplus}(x)
ln⁡(1+ex)\ln(1+e^x); the last form is the overflow-free way to evaluate it
∂ℓ∂z=σ(z)−y∂2ℓ∂z2=σ(z)(1−σ(z))∈(0,14]\frac{\partial\ell}{\partial z}=\sigma(z)-y\qquad\qquad\frac{\partial^2\ell}{\partial z^2}=\sigma(z)\big(1-\sigma(z)\big)\in(0,\tfrac14]
σ(z)−y\sigma(z)-y
prediction minus target: bounded in (−1,1)(-1,1), and ≈∓1\approx\mp1 when the model is confidently wrong
σ(1−σ)\sigma(1-\sigma)
positive curvature, so BCE is convex in zz (and in the weights of a logistic regression)

For a soft target the minimum is at σ(z∗)=y\sigma(z^*)=y, i.e. z∗=logit⁡(y)z^*=\operatorname{logit}(y). A hard target y=1y=1 has its minimum at z→+∞z\to+\infty: the loss keeps decreasing as the logit grows, which is why unregularized classifiers keep growing their weights on separable data.

At a glance

Task
binary classification; multi-label classification (one BCE per label)
Output layer
one logit per binary decision, no sigmoid inside the model when training
Range
[0,∞)[0,\infty); ln⁡2=0.693\ln 2=0.693 at p=0.5p=0.5
Gradient w.r.t. z
σ(z)−y\sigma(z)-y; with pos_weight ww: σ(z)(wy+1−y)−wy\sigma(z)(wy+1-y)-wy
Probabilistic model
y∼Bernoulli(σ(z))y\sim\mathrm{Bernoulli}(\sigma(z))
PyTorch
nn.BCEWithLogitsLoss, F.binary_cross_entropy_with_logits; nn.BCELoss only for probabilities
Keras
keras.losses.BinaryCrossentropy(from_logits=True)
loss ℓ(z) = w·y·softplus(−z) + (1−y)·softplus(z)02468y=1 branch ×wy=0 branch∂ℓ/∂z = σ(z)(w·y + 1 − y) − w·y1.0−1.0−8−4048logit zlogit z−2.00p = σ(z)0.1192target y1.00pos_weight w1.00with_logits loss2.1269∂ℓ/∂z−0.8808σ → BCELoss (f32)2.1269p in float32okbest z* (min ℓ)+∞σ(z*)1z* = logit(y) + ln w
fig.Top: the loss as a function of the logit for the chosen target (solid), with the weighted y = 1 branch and the y = 0 branch dashed. Bottom: the gradient. At the default z = −2, y = 1 the loss is 2.1269 and the gradient −0.8808: a wrong, confident logit gets almost the full unit push. Slide y to a soft value and the minimum z* moves to logit(y); at y = 0.2 it sits at −1.386. Raise pos_weight and the y = 1 branch is scaled by w, moving z* by ln w: the '20% positives, w = 4' preset puts z* at exactly 0. 'Overflow test' sets z = 20, y = 0: the logit form returns 20.0000, but sigmoid rounded to float32 is exactly 1 and BCELoss reports its clamp value 100.

Why the logit form is not optional

The two-step recipe p = torch.sigmoid(z) then nn.BCELoss()(p, y) is mathematically identical and numerically broken. In float32, σ(z)\sigma(z) rounds to exactly 1 once e−ze^{-z} is below half a unit in the last place of 1, which happens near z≈16.6z\approx16.6. From there on, a negative example has 1−p=01-p=0 and ln⁡0=−∞\ln 0=-\infty. PyTorch's BCELoss clamps each log term at −100-100 to avoid infinities, so the loss reads 100 instead of the true value, and the gradient computed from the rounded probability no longer resembles the true σ(z)−y\sigma(z)-y.

logit zexact softplus(z)σ → BCELoss (float32)σ(z) in float32
1010.00009.99960.99995458
1515.000014.84380.99999964
1616.000015.94240.99999988
1717.0000100 (clamped)exactly 1
3030.0000100 (clamped)exactly 1
Negative example (y = 0), loss in nats: exact logit form vs sigmoid rounded to float32 followed by BCELoss (computed by the figure's float32 emulation)

The positive side is safe in the naive version only because float32 can represent tiny probabilities (σ(−30)≈9.4×10−14\sigma(-30)\approx9.4\times10^{-14}) but cannot represent numbers just below 1. The logit form avoids the asymmetry: max⁡(z,0)−yz+ln⁡(1+e−∣z∣)\max(z,0)-yz+\ln(1+e^{-|z|}) only ever exponentiates a non-positive number. It also lets autograd differentiate the stable expression, giving the clean σ(z)−y\sigma(z)-y instead of a ratio of rounded quantities.

Class imbalance: pos_weight

When positives are rare, the average loss is dominated by easy negatives and a model can score well by predicting "no" everywhere. pos_weight multiplies the positive term only:

ℓw(z,y)=−[w yln⁡σ(z)+(1−y)ln⁡(1−σ(z))],z∗=ln⁡w π1−π\ell_w(z,y)=-\big[w\,y\ln\sigma(z)+(1-y)\ln(1-\sigma(z))\big],\qquad z^*=\ln\frac{w\,\pi}{1-\pi}
ww
pos_weight; w>1w>1 raises recall, w<1w<1 raises precision
π\pi
fraction of positives among examples that look alike (the soft target the model sees on average)
z∗z^*
best constant logit: logit⁡(π)+ln⁡w\operatorname{logit}(\pi)+\ln w

The minimizer shows exactly what the weight does: it shifts the optimal logit by ln⁡w\ln w. With 20% positives the unweighted optimum is σ(z∗)=0.2\sigma(z^*)=0.2, z∗=−1.386z^*=-1.386. PyTorch's documentation suggests w=#neg/#posw=\#\text{neg}/\#\text{pos}, here 0.8/0.2=40.8/0.2=4, which moves z∗z^* to ln⁡(4⋅0.2/0.8)=0\ln(4\cdot0.2/0.8)=0, so the model outputs 0.5 where positives are 20%. That is the point of weighting, and also its cost: the outputs are no longer calibrated probabilities. To recover P(y=1∣x)P(y=1\mid x) subtract ln⁡w\ln w from the logit before the sigmoid.

Multi-label: one BCE per label

In multi-label classification each example can have any subset of LL labels (a photo can contain both a dog and a beach). The model outputs LL logits, each label gets its own sigmoid, and the loss is the mean of LL independent BCE terms. Unlike softmax, the probabilities do not compete and need not sum to 1.

L(x)=1L∑j=1L[softplus⁡(zj)−yjzj],y∈{0,1}L\mathcal L(x)=\frac1L\sum_{j=1}^{L}\Big[\operatorname{softplus}(z_j)-y_jz_j\Big],\qquad y\in\{0,1\}^L
zjz_j
logit of label jj
yy
multi-hot target vector (float), e.g. (1,0,1)(1,0,1)

Example: logits (1.2, −0.4, 2.5)(1.2,\,-0.4,\,2.5) with target (1,0,1)(1,0,1) give p=(0.7685, 0.4013, 0.9241)p=(0.7685,\,0.4013,\,0.9241), which sum to 2.094, and per-label losses (0.2633, 0.5130, 0.0789)(0.2633,\,0.5130,\,0.0789) with mean 0.2851. Each label has its own decision threshold, tuned on validation data.

Worked example

logit zyp = σ(z)loss∂ℓ/∂z = p − yloss, w = 3∂ℓ/∂z, w = 3
2.010.88080.1269−0.11920.3808−0.3576
−1.010.26891.3133−0.73113.9398−2.1932
0.500.62250.97410.62250.97410.6225
−3.000.04740.04860.04740.04860.0474
A batch of four examples, mean reduction (checked numerically in Node). Rightmost columns use pos_weight w = 3.

Reading the table

  1. 1Example 2 is a missed positive: ℓ=softplus⁡(−1)=ln⁡(1+e)=1.3133\ell=\operatorname{softplus}(-1)=\ln(1+e)=1.3133 and the gradient 0.2689−1=−0.73110.2689-1=-0.7311 pushes the logit up hard.
  2. 2Mean loss =(0.1269+1.3133+0.9741+0.0486)/4=0.6157=(0.1269+1.3133+0.9741+0.0486)/4=0.6157. The per-example gradients of the mean are the column divided by 4.
  3. 3With w=3w=3 only the positive rows change: their loss and gradient are multiplied by exactly 3 (1.3133→3.93981.3133\to3.9398, −0.7311→−2.1932-0.7311\to-2.1932). The mean rises to 1.3358, and the negatives' share of the gradient falls.
  4. 4Example 4 is an easy negative. Its gradient 0.0474 is small but never exactly zero, so easy examples still pull a little. Focal loss (Lin et al., 2017) multiplies by (1−pt)γ(1-p_t)^\gamma to suppress them further.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3z = torch.tensor([2.0, -1.0, 0.5, -3.0], requires_grad=True) # raw logits
4y = torch.tensor([1.0, 1.0, 0.0, 0.0]) # float targets
5
6loss = F.binary_cross_entropy_with_logits(z, y) # 0.6157
7loss.backward()
8print(z.grad * 4) # sigmoid(z) - y: [-0.1192, -0.7311, 0.6225, 0.0474]
9
10# class imbalance: scale the positive term (suggested: #neg / #pos)
11crit = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(3.0))
12print(crit(z, y)) # 1.3358
13
14# the naive two-step version saturates in float32
15big = torch.tensor([20.0])
16print(nn.BCELoss()(torch.sigmoid(big), torch.tensor([0.0]))) # 100.0 (clamped)
17print(F.binary_cross_entropy_with_logits(big, torch.tensor([0.0]))) # 20.0
18
19# multi-label: L logits, multi-hot float targets, one pos_weight per label
20logits = torch.tensor([[1.2, -0.4, 2.5]])
21target = torch.tensor([[1.0, 0.0, 1.0]])
22print(nn.BCEWithLogitsLoss()(logits, target)) # 0.2851
23probs = torch.sigmoid(logits) # independent: need not sum to 1
24pred = probs > 0.5 # per-label thresholds in practice
BCE from logits, pos_weight, and multi-label in PyTorch

Use when

  • +Proper scoring rule: the expected loss is minimized by the true probability, so outputs are calibrated (without pos_weight)
  • +Gradient σ(z)−y\sigma(z)-y is bounded and does not vanish when the model is confidently wrong
  • +Convex in the logit; logistic regression with BCE has a single optimum
  • +Handles multi-label tasks and soft targets with no change

Avoid when

  • −Unbounded: a single mislabelled example with a confident logit contributes a huge loss
  • −On separable data the optimum is at ∣z∣→∞|z|\to\infty, so it needs weight decay or early stopping
  • −Easy negatives still dominate the sum under heavy imbalance; pos_weight helps but de-calibrates
  • −The probability form (BCELoss) is numerically fragile; only the logit form is safe

Check yourself

A detector is trained with BCEWithLogitsLoss and pos_weight = 9 because 10% of examples are positive. On inputs where the true positive rate is 10%, what probability does it output?

Check yourself

An image can be tagged with any of 20 attributes. Which output head and loss fit?

Remember

  • ✓BCE is the Bernoulli NLL: −[yln⁡p+(1−y)ln⁡(1−p)]-[y\ln p+(1-y)\ln(1-p)], which on the logit is softplus⁡(z)−yz\operatorname{softplus}(z)-yz with gradient σ(z)−y\sigma(z)-y.
  • ✓Always pass logits to BCEWithLogitsLoss: in float32, σ(z)\sigma(z) is exactly 1 above z≈16.6z\approx16.6 and BCELoss then reports its clamp value 100.
  • ✓pos_weight =w=w multiplies the positive term and moves the optimal logit by ln⁡w\ln w; it trades calibration for recall.
  • ✓Multi-label = one independent sigmoid + BCE per label; probabilities need not sum to 1.
  • ✓Soft targets work directly: the minimum is at σ(z)=y\sigma(z)=y.
← back to Deep Learning roadmap