Definition
CTC loss is the negative log-likelihood of a target label sequence under Connectionist Temporal Classification, where is the total probability of every frame-level path that collapses to . It is computed exactly, in time, by a forward–backward dynamic programme over the target with blanks inserted.
Connectionist Temporal Classification explains the model: a per-frame softmax over labels plus a blank , the collapse map (merge repeats, then drop blanks), and . This article is about computing that sum and its gradient, and about using nn.CTCLoss without the usual mistakes. The sum has terms for a repeat-free target (about for 3 labels in 100 frames), so it cannot be enumerated. The recursion below is from Graves, Fernández, Gomez and Schmidhuber (ICML 2006). It is the same forward–backward idea as for hidden Markov models.
The extended label sequence
Insert a blank at the start, between every pair of labels and at the end: becomes . A target of labels has states. Every alignment is a monotone walk through that spends at least one frame on each label and may skip any blank, except one that sits between two equal labels.
Allowed moves from frame t−1 to frame t
- 1Stay in state : a label or blank held for another frame (merged by ).
- 2Advance from to : blank → label or label → blank.
- 3Skip from to over a blank, allowed only if and . Two different labels may touch (
ab); two equal ones may not, oraawould merge into onea. - 4Start in state 1 (leading blank) or state 2 (first label). End in state (trailing blank) or (last label).
Forward (α) and backward (β) recursions
- total probability of all path prefixes that are valid starts of an alignment and sit in state at frame
- softmax output for symbol at frame
- 1 if and , else 0; all other
- , the trailing blank; is the last label
The backward variable runs the same recursion from the end, with , . With this convention both and include the frame- output, so their product counts it twice. Dividing once gives the total mass of all complete alignments that pass through :
- , the posterior probability that frame is aligned to state ;
Gradient with respect to the logits
Let be the pre-softmax logits, . Differentiating through the recursion and the softmax (Graves et al., 2006, eq. 16) gives:
- posterior probability that frame emits symbol , summed over every state carrying (all blank states for )
- what the network currently predicts for frame t
This is the softmax cross-entropy gradient with soft frame targets . The targets are not given; they are the model's own posterior over alignments, recomputed every step, which makes CTC training an EM-like loop: infer a soft alignment (E), fit the frame classifier to it (M). Each component lies in and each frame's gradient sums to 0.
At a glance
- Loss
- ; 0 only if one alignment has probability 1; if no alignment fits in frames
- States
- (blank-padded target)
- Cost
- time and memory for α and β, per utterance; frames are sequential, the batch and states parallelise
- Feasibility
- , = number of adjacent equal labels (each needs a separating blank)
- Gradient
- per frame: softmax minus posterior occupancy, bounded, sums to 0
- Numerics
- run in log space with log-sum-exp; in probability space α underflows float32 after ~126 frames at ()
- PyTorch
nn.CTCLoss(blank=0, reduction="mean", zero_infinity=False), inputs(log_probs, targets, input_lengths, target_lengths)
Interactive: the α lattice
Worked example
Alphabet , target , , so . The frame posteriors are , , over .
| t | s=1 ε | s=2 a | s=3 ε | s=4 b | s=5 ε |
|---|---|---|---|---|---|
| 1 | 0.5000 | 0.4000 | 0 | 0 | 0 |
| 2 | 0 | ||||
| 3 | 0.0600 | 0.0420 | 0.1560 |
So and . Check by brute force: the 5 alignments are aab (0.0600), abb (0.0800), -ab (0.0750), a-b (0.0600) and ab- (0.0640), which also sum to 0.3390. The skip into at (from the label straight to ) is the term that makes ab adjacency possible.
| t | β_t (s=1…5) | Σγ for ε, a, b | ∂L/∂u = y − Σγ |
|---|---|---|---|
| 1 | 0.0750, 0.2640, 0.2550, 0.0480, 0.0600 | 0.2212, 0.7788, 0 | (+0.2788, −0.3788, +0.1000) |
| 2 | 0, 0.1500, 0.1500, 0.3600, 0.1200 | 0.1770, 0.3982, 0.4248 | (+0.1230, −0.0982, −0.0248) |
| 3 | 0, 0, 0, 0.5000, 0.4000 | 0.1888, 0, 0.8112 | (+0.2112, +0.1000, −0.3112) |
At the occupancy is : 78% of the alignment mass has "a" in frame 1. The network gives "a" only 0.4, so its logit is pushed up (), and blank and "b" are pushed down. At , almost matches its posterior 0.4248, so gets only a small push (). Every row of the gradient sums to 0.
Log space, not rescaling
α is a product of probabilities, so it underflows quickly. Graves (2006) rescaled α and β at each frame by their sum and accumulated the log of the scale factors. Modern implementations (PyTorch, warp-ctc, cuDNN, Optax) instead store and replace every sum by log-sum-exp, with , and for unreachable states. Inputs are log-probabilities from log_softmax, so the network output never passes through a raw probability.
In PyTorch: nn.CTCLoss
1import torch, torch.nn as nn, torch.nn.functional as F23ctc = nn.CTCLoss(blank=0, reduction="mean", zero_infinity=True)45# a batch: T frames (after any downsampling!), N utterances, C = labels + 1 blank6T, N, C = 50, 4, 287logits = torch.randn(T, N, C, requires_grad=True) # time-major: (T, N, C)8log_probs = logits.log_softmax(dim=-1) # log-probs, NOT softmax probs9target_lengths = torch.tensor([12, 7, 20, 3])10targets = torch.randint(1, C, (N, int(target_lengths.max()))) # padded (N, S); never use 0 = blank11input_lengths = torch.tensor([50, 41, 50, 18]) # valid frames per utterance, each <= T12loss = ctc(log_probs, targets, input_lengths, target_lengths)13loss.backward()1415# the worked example: T=3, alphabet (blank, a, b), target "ab"16y = torch.tensor([[0.5, 0.4, 0.1], [0.3, 0.3, 0.4], [0.4, 0.1, 0.5]])17u = y.log().unsqueeze(1).requires_grad_() # (T=3, N=1, C=3) logits18lp = u.log_softmax(-1)19nll = F.ctc_loss(lp, torch.tensor([[1, 2]]), torch.tensor([3]), torch.tensor([2]), reduction="sum")20nll.backward()21print(nll) # 1.0818 = -log 0.33922print(u.grad.squeeze(1)) # [[ 0.2788, -0.3788, 0.1000],23 # [ 0.1230, -0.0982, -0.0248],24 # [ 0.2112, 0.1000, -0.3112]]| Argument | Shape / value | Meaning |
|---|---|---|
log_probs | (T, N, C) or (T, C) | log-softmax outputs, time first; float32 is safest |
targets | (N, S) padded, or 1-D of length Σ target_lengths | label indices in 1…C−1 (anything except blank) |
input_lengths | (N,) | valid frames per sequence, ≤ T; padding frames beyond it are ignored |
target_lengths | (N,) | true length of each target; padding in targets beyond it is ignored |
blank | int, default 0 | index of the blank in the C classes |
reduction | 'mean' (default), 'sum', 'none' | mean divides each loss by its target length, then averages over the batch; none returns (N,) |
zero_infinity | bool, default False | replace infinite losses (infeasible T) and their gradients by 0 |
Use when
- +Exact marginal likelihood over all alignments in O(T·U), no frame labels required.
- +Simple, bounded gradient: softmax minus posterior occupancy, like cross-entropy with soft targets.
- +Fully supported and fast in every framework (PyTorch native and cuDNN, TensorFlow, Optax).
- +Works as an auxiliary loss on top of attention or transducer models to enforce monotonic alignment.
Avoid when
- −Infinite for infeasible lengths; one bad sample can NaN a whole batch unless lengths are checked.
- −Sequential over frames: the α/β recursion cannot be parallelised over time.
- −The loss value depends on reduction and length normalisation, which makes comparisons across setups tricky.
- −Inherits CTC's conditional-independence assumption: the loss itself has no notion of label context.
Check yourself
For the target "abb", which skip transition is forbidden in the extended sequence ?
Check yourself
At some frame the network predicts , and the alignment posterior puts total occupancy on blank states. What is , and which way does gradient descent move the blank logit?
Remember
- ✓Pad the target with blanks: has states; moves are stay, advance, or skip a blank between different labels.
- ✓; .
- ✓ at every : a built-in correctness check.
- ✓Gradient w.r.t. logits: , cross-entropy against the model's own soft alignment.
- ✓Compute in log space with log-sum-exp; probability-space α underflows within a few hundred frames.
- ✓
nn.CTCLoss: time-major log-probs (T, N, C), blank never in targets,input_lengthsafter subsampling,'mean'divides by target length. - ✓Inf loss means for some sample: fix the lengths, use
zero_infinityonly as a guard.