Definition
Binary cross-entropy (BCE, log loss) scores a predicted probability of the positive class against a target (or a soft target ) as . It is the negative log-likelihood of under a Bernoulli distribution with parameter .
BCE is the loss for any yes/no output: spam or not, click or not, real or fake in a GAN discriminator, and each label of a multi-label tagger. The Loss functions overview showed that it comes from a Bernoulli likelihood and that its logit gradient is . This article covers how it is actually used: the logit form that frameworks compute, why that form is needed numerically, how pos_weight handles class imbalance, and how one BCE per label gives multi-label classification.
The loss is older than neural networks. It is the log-likelihood of logistic regression (Cox, 1958), whose log-odds link Berkson had called the logit in 1944. The name "cross-entropy" comes from information theory: for a target distribution and a model it is exactly the cross-entropy . In 1988 Baum and Wilczek, and independently Solla, Levin and Fleisher, proposed it for training neural networks with probabilistic outputs in place of squared error.
Formula on the logit
Networks output a logit (an unbounded real number), not a probability. Substituting and using and turns the loss into a function of that never takes the log of a rounded probability:
- the logit, ; the raw output of the last linear layer
- target: 0 or 1, or any value in for a soft label
- ; the last form is the overflow-free way to evaluate it
- prediction minus target: bounded in , and when the model is confidently wrong
- positive curvature, so BCE is convex in (and in the weights of a logistic regression)
For a soft target the minimum is at , i.e. . A hard target has its minimum at : the loss keeps decreasing as the logit grows, which is why unregularized classifiers keep growing their weights on separable data.
At a glance
- Task
- binary classification; multi-label classification (one BCE per label)
- Output layer
- one logit per binary decision, no sigmoid inside the model when training
- Range
- ; at
- Gradient w.r.t. z
- ; with
pos_weight: - Probabilistic model
- PyTorch
nn.BCEWithLogitsLoss,F.binary_cross_entropy_with_logits;nn.BCELossonly for probabilities- Keras
keras.losses.BinaryCrossentropy(from_logits=True)
Why the logit form is not optional
The two-step recipe p = torch.sigmoid(z) then nn.BCELoss()(p, y) is mathematically identical and numerically broken. In float32, rounds to exactly 1 once is below half a unit in the last place of 1, which happens near . From there on, a negative example has and . PyTorch's BCELoss clamps each log term at to avoid infinities, so the loss reads 100 instead of the true value, and the gradient computed from the rounded probability no longer resembles the true .
| logit z | exact softplus(z) | σ → BCELoss (float32) | σ(z) in float32 |
|---|---|---|---|
| 10 | 10.0000 | 9.9996 | 0.99995458 |
| 15 | 15.0000 | 14.8438 | 0.99999964 |
| 16 | 16.0000 | 15.9424 | 0.99999988 |
| 17 | 17.0000 | 100 (clamped) | exactly 1 |
| 30 | 30.0000 | 100 (clamped) | exactly 1 |
The positive side is safe in the naive version only because float32 can represent tiny probabilities () but cannot represent numbers just below 1. The logit form avoids the asymmetry: only ever exponentiates a non-positive number. It also lets autograd differentiate the stable expression, giving the clean instead of a ratio of rounded quantities.
Class imbalance: pos_weight
When positives are rare, the average loss is dominated by easy negatives and a model can score well by predicting "no" everywhere. pos_weight multiplies the positive term only:
pos_weight; raises recall, raises precision- fraction of positives among examples that look alike (the soft target the model sees on average)
- best constant logit:
The minimizer shows exactly what the weight does: it shifts the optimal logit by . With 20% positives the unweighted optimum is , . PyTorch's documentation suggests , here , which moves to , so the model outputs 0.5 where positives are 20%. That is the point of weighting, and also its cost: the outputs are no longer calibrated probabilities. To recover subtract from the logit before the sigmoid.
Multi-label: one BCE per label
In multi-label classification each example can have any subset of labels (a photo can contain both a dog and a beach). The model outputs logits, each label gets its own sigmoid, and the loss is the mean of independent BCE terms. Unlike softmax, the probabilities do not compete and need not sum to 1.
- logit of label
- multi-hot target vector (float), e.g.
Example: logits with target give , which sum to 2.094, and per-label losses with mean 0.2851. Each label has its own decision threshold, tuned on validation data.
Worked example
| logit z | y | p = σ(z) | loss | ∂ℓ/∂z = p − y | loss, w = 3 | ∂ℓ/∂z, w = 3 |
|---|---|---|---|---|---|---|
| 2.0 | 1 | 0.8808 | 0.1269 | −0.1192 | 0.3808 | −0.3576 |
| −1.0 | 1 | 0.2689 | 1.3133 | −0.7311 | 3.9398 | −2.1932 |
| 0.5 | 0 | 0.6225 | 0.9741 | 0.6225 | 0.9741 | 0.6225 |
| −3.0 | 0 | 0.0474 | 0.0486 | 0.0474 | 0.0486 | 0.0474 |
Reading the table
- 1Example 2 is a missed positive: and the gradient pushes the logit up hard.
- 2Mean loss . The per-example gradients of the mean are the column divided by 4.
- 3With only the positive rows change: their loss and gradient are multiplied by exactly 3 (, ). The mean rises to 1.3358, and the negatives' share of the gradient falls.
- 4Example 4 is an easy negative. Its gradient 0.0474 is small but never exactly zero, so easy examples still pull a little. Focal loss (Lin et al., 2017) multiplies by to suppress them further.
1import torch, torch.nn as nn, torch.nn.functional as F23z = torch.tensor([2.0, -1.0, 0.5, -3.0], requires_grad=True) # raw logits4y = torch.tensor([1.0, 1.0, 0.0, 0.0]) # float targets56loss = F.binary_cross_entropy_with_logits(z, y) # 0.61577loss.backward()8print(z.grad * 4) # sigmoid(z) - y: [-0.1192, -0.7311, 0.6225, 0.0474]910# class imbalance: scale the positive term (suggested: #neg / #pos)11crit = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(3.0))12print(crit(z, y)) # 1.33581314# the naive two-step version saturates in float3215big = torch.tensor([20.0])16print(nn.BCELoss()(torch.sigmoid(big), torch.tensor([0.0]))) # 100.0 (clamped)17print(F.binary_cross_entropy_with_logits(big, torch.tensor([0.0]))) # 20.01819# multi-label: L logits, multi-hot float targets, one pos_weight per label20logits = torch.tensor([[1.2, -0.4, 2.5]])21target = torch.tensor([[1.0, 0.0, 1.0]])22print(nn.BCEWithLogitsLoss()(logits, target)) # 0.285123probs = torch.sigmoid(logits) # independent: need not sum to 124pred = probs > 0.5 # per-label thresholds in practiceUse when
- +Proper scoring rule: the expected loss is minimized by the true probability, so outputs are calibrated (without
pos_weight) - +Gradient is bounded and does not vanish when the model is confidently wrong
- +Convex in the logit; logistic regression with BCE has a single optimum
- +Handles multi-label tasks and soft targets with no change
Avoid when
- −Unbounded: a single mislabelled example with a confident logit contributes a huge loss
- −On separable data the optimum is at , so it needs weight decay or early stopping
- −Easy negatives still dominate the sum under heavy imbalance;
pos_weighthelps but de-calibrates - −The probability form (
BCELoss) is numerically fragile; only the logit form is safe
Check yourself
A detector is trained with BCEWithLogitsLoss and pos_weight = 9 because 10% of examples are positive. On inputs where the true positive rate is 10%, what probability does it output?
Check yourself
An image can be tagged with any of 20 attributes. Which output head and loss fit?
Remember
- ✓BCE is the Bernoulli NLL: , which on the logit is with gradient .
- ✓Always pass logits to
BCEWithLogitsLoss: in float32, is exactly 1 above andBCELossthen reports its clamp value 100. - ✓
pos_weightmultiplies the positive term and moves the optimal logit by ; it trades calibration for recall. - ✓Multi-label = one independent sigmoid + BCE per label; probabilities need not sum to 1.
- ✓Soft targets work directly: the minimum is at .