CTC loss

−log p(y|x) summed over every alignment by a forward–backward pass over the blank-padded target (2U+1 states). The gradient on each logit is softmax minus posterior occupancy; in PyTorch it is nn.CTCLoss on (T, N, C) log-probs.

14 min read991 wordsadvanced1 viewsupdated 2026-10-11

Definition

CTC loss is the negative log-likelihood L=−log⁡p(y∣x)\mathcal L=-\log p(y\mid x) of a target label sequence yy under Connectionist Temporal Classification, where p(y∣x)p(y\mid x) is the total probability of every frame-level path that collapses to yy. It is computed exactly, in O(T⋅U)O(T\cdot U) time, by a forward–backward dynamic programme over the target with blanks inserted.

Connectionist Temporal Classification explains the model: a per-frame softmax over labels plus a blank ε\varepsilon, the collapse map B\mathcal B (merge repeats, then drop blanks), and p(y∣x)=∑π∈B−1(y)∏tyt(πt)p(y\mid x)=\sum_{\pi\in\mathcal B^{-1}(y)}\prod_t y_t(\pi_t). This article is about computing that sum and its gradient, and about using nn.CTCLoss without the usual mistakes. The sum has (T+U2U)\binom{T+U}{2U} terms for a repeat-free target (about 1.4×1091.4\times10^9 for 3 labels in 100 frames), so it cannot be enumerated. The recursion below is from Graves, Fernández, Gomez and Schmidhuber (ICML 2006). It is the same forward–backward idea as for hidden Markov models.

The extended label sequence

Insert a blank at the start, between every pair of labels and at the end: y=(a,b)y=(a,b) becomes l′=(ε,a,ε,b,ε)l'=(\varepsilon,a,\varepsilon,b,\varepsilon). A target of UU labels has S=2U+1S=2U+1 states. Every alignment is a monotone walk through l′l' that spends at least one frame on each label and may skip any blank, except one that sits between two equal labels.

Allowed moves from frame t−1 to frame t

  1. 1Stay in state ss: a label or blank held for another frame (merged by B\mathcal B).
  2. 2Advance from s−1s-1 to ss: blank → label or label → blank.
  3. 3Skip from s−2s-2 to ss over a blank, allowed only if ls′≠εl'_s\ne\varepsilon and ls′≠ls−2′l'_s\ne l'_{s-2}. Two different labels may touch (ab); two equal ones may not, or aa would merge into one a.
  4. 4Start in state 1 (leading blank) or state 2 (first label). End in state SS (trailing blank) or S−1S-1 (last label).

Forward (α) and backward (β) recursions

αt(s)=(αt−1(s)+αt−1(s−1)+1[skip] αt−1(s−2)) yt(ls′),α1(1)=y1(ε),    α1(2)=y1(l2′)\alpha_t(s)=\Big(\alpha_{t-1}(s)+\alpha_{t-1}(s-1)+\mathbb 1[\text{skip}]\,\alpha_{t-1}(s-2)\Big)\,y_t(l'_s),\qquad \alpha_1(1)=y_1(\varepsilon),\;\;\alpha_1(2)=y_1(l'_2)
αt(s)\alpha_t(s)
total probability of all path prefixes π1:t\pi_{1:t} that are valid starts of an alignment and sit in state ss at frame tt
yt(k)y_t(k)
softmax output for symbol kk at frame tt
1[skip]\mathbb 1[\text{skip}]
1 if ls′≠εl'_s\ne\varepsilon and ls′≠ls−2′l'_s\ne l'_{s-2}, else 0; all other α1(s)=0\alpha_1(s)=0
p(y∣x)=αT(S)+αT(S−1),LCTC=−log⁡p(y∣x)p(y\mid x)=\alpha_T(S)+\alpha_T(S-1),\qquad \mathcal L_{\text{CTC}}=-\log p(y\mid x)
SS
2U+12U+1, the trailing blank; S−1S-1 is the last label

The backward variable runs the same recursion from the end, βt(s)=(βt+1(s)+βt+1(s+1)+1[skip into s+2] βt+1(s+2)) yt(ls′)\beta_t(s)=\big(\beta_{t+1}(s)+\beta_{t+1}(s+1)+\mathbb 1[\text{skip into }s+2]\,\beta_{t+1}(s+2)\big)\,y_t(l'_s) with βT(S)=yT(ε)\beta_T(S)=y_T(\varepsilon), βT(S−1)=yT(lS−1′)\beta_T(S-1)=y_T(l'_{S-1}). With this convention both αt(s)\alpha_t(s) and βt(s)\beta_t(s) include the frame-tt output, so their product counts it twice. Dividing once gives the total mass of all complete alignments that pass through (t,s)(t,s):

αt(s) βt(s)yt(ls′)=∑π∈B−1(y):  π in state s at tp(π∣x),∑s=1Sαt(s)βt(s)yt(ls′)=p(y∣x)    for every t\frac{\alpha_t(s)\,\beta_t(s)}{y_t(l'_s)}=\sum_{\pi\in\mathcal B^{-1}(y):\;\pi\text{ in state }s\text{ at }t} p(\pi\mid x),\qquad \sum_{s=1}^{S}\frac{\alpha_t(s)\beta_t(s)}{y_t(l'_s)}=p(y\mid x)\;\;\text{for every }t
γt(s)\gamma_t(s)
αt(s)βt(s)/(yt(ls′) p(y∣x))\alpha_t(s)\beta_t(s)/\big(y_t(l'_s)\,p(y\mid x)\big), the posterior probability that frame tt is aligned to state ss; ∑sγt(s)=1\sum_s\gamma_t(s)=1

Gradient with respect to the logits

Let ut(k)u_t(k) be the pre-softmax logits, yt=softmax⁡(ut)y_t=\operatorname{softmax}(u_t). Differentiating −log⁡p-\log p through the recursion and the softmax (Graves et al., 2006, eq. 16) gives:

∂L∂ut(k)=yt(k)−1p(y∣x)∑s: ls′=kαt(s)βt(s)yt(k)=yt(k)−∑s: ls′=kγt(s)\frac{\partial\mathcal L}{\partial u_t(k)}=y_t(k)-\frac{1}{p(y\mid x)}\sum_{s:\,l'_s=k}\frac{\alpha_t(s)\beta_t(s)}{y_t(k)}=y_t(k)-\sum_{s:\,l'_s=k}\gamma_t(s)
∑s:ls′=kγt(s)\sum_{s:l'_s=k}\gamma_t(s)
posterior probability that frame tt emits symbol kk, summed over every state carrying kk (all blank states for k=εk=\varepsilon)
yt(k)y_t(k)
what the network currently predicts for frame t

This is the softmax cross-entropy gradient p−qp-q with soft frame targets qt(k)=∑s:ls′=kγt(s)q_t(k)=\sum_{s:l'_s=k}\gamma_t(s). The targets are not given; they are the model's own posterior over alignments, recomputed every step, which makes CTC training an EM-like loop: infer a soft alignment (E), fit the frame classifier to it (M). Each component lies in [−1,1][-1,1] and each frame's gradient sums to 0.

At a glance

Loss
−log⁡p(y∣x)≥0-\log p(y\mid x)\ge0; 0 only if one alignment has probability 1; +∞+\infty if no alignment fits in TT frames
States
S=2U+1S=2U+1 (blank-padded target)
Cost
O(T⋅S)O(T\cdot S) time and memory for α and β, per utterance; frames are sequential, the batch and states parallelise
Feasibility
T≥U+rT\ge U+r, rr = number of adjacent equal labels (each needs a separating blank)
Gradient
yt−qty_t-q_t per frame: softmax minus posterior occupancy, bounded, sums to 0
Numerics
run in log space with log-sum-exp; in probability space α underflows float32 after ~126 frames at y≈0.5y\approx0.5 (0.5126≈1.2×10−380.5^{126}\approx1.2\times10^{-38})
PyTorch
nn.CTCLoss(blank=0, reduction="mean", zero_infinity=False), inputs (log_probs, targets, input_lengths, target_lengths)

Interactive: the α lattice

log α_t(s): all prefixes ending in state s at frame tt=1t=2t=3t=4t=5t=61:ε2:a3:ε4:b5:ε6:c7:ε−0.5−1.9−4.5···−1.9−1.2−1.8−3.4···−3.3−3.6−3.6···−3.6−2.2−1.9−2.8···−6.2−4.1−2.9···−4.9−4.1−3.9−3.1···−6.8−5.2−4.7grey = off every complete path · violet = skip over ε (only between different labels)cell t=4, s=4 (b) α3(4) 0.1112+ α3(3) 0.0264+ α3(2) skip 0.1620× y_4(b) = 0.514= 0.1541frame 4: ∂L/∂u = y − Σγε0.15+0.044a0.20+0.143b0.51−0.263c0.13+0.076columns: y_t(k), gradienttarget y = “abc” → l′ has 2·3+1 = 7 states · needs T ≥ 3 · T = 6p(y|x) = α_T(S) + α_T(S−1) = 0.0537 · loss −log p = 2.9252check: Σₛ αβ/y = p at every frame (max |log error| 9e-16)train: watch γ collapse onto a single alignment as the loss falls
targetview
trainclick any cell to see its recursion
fig.Rows are the 2U+1 states of the blank-extended target, columns are frames, and the frame posteriors come from fixed random logits. Default: target "abc", T = 6, so 7 states and T ≥ 3. p(y|x) = α₆(7) + α₆(6) = 0.0537 and the loss is 2.9252. Click a cell to see its recursion: α₄(4) for the label b is (α₃(4) + α₃(3) + α₃(2)) × y₄(b) = (0.1112 + 0.0264 + 0.1620) × 0.514 ≈ 0.154. The skip from α₃(2) is allowed because a ≠ b. Grey cells lie on no complete path. Violet edges are skips; choose "abb" and the skip into the second b disappears. Drop T to 2 and every target becomes infeasible (loss = inf). Switch to β or γ and check that Σₛ αβ/y equals p at every frame. Train with free logits: the loss falls to 0.4309 after 10 steps and 0.0402 after 100, and γ concentrates on one alignment (argmax path εaabbc).

Worked example

Alphabet {ε,a,b}\{\varepsilon,a,b\}, target y=(a,b)y=(a,b), T=3T=3, so l′=(ε,a,ε,b,ε)l'=(\varepsilon,a,\varepsilon,b,\varepsilon). The frame posteriors are y1=(0.5,0.4,0.1)y_1=(0.5,0.4,0.1), y2=(0.3,0.3,0.4)y_2=(0.3,0.3,0.4), y3=(0.4,0.1,0.5)y_3=(0.4,0.1,0.5) over (ε,a,b)(\varepsilon,a,b).

ts=1 εs=2 as=3 εs=4 bs=5 ε
10.50000.4000000
20.5⋅0.3=0.15000.5\cdot0.3=0.1500(0.4+0.5)⋅0.3=0.2700(0.4+0.5)\cdot0.3=0.2700(0+0.4)⋅0.3=0.1200(0+0.4)\cdot0.3=0.1200(0+0+0.4)⋅0.4=0.1600(0+0+0.4)\cdot0.4=0.16000
30.06000.04200.1560(0.16+0.12+0.27)⋅0.5=0.2750(0.16+0.12+0.27)\cdot0.5=0.2750(0+0.16)⋅0.4=0.0640(0+0.16)\cdot0.4=0.0640
α_t(s): forward lattice (exact to 4 decimals)

So p(y∣x)=α3(5)+α3(4)=0.0640+0.2750=0.3390p(y\mid x)=\alpha_3(5)+\alpha_3(4)=0.0640+0.2750=0.3390 and L=−log⁡0.339=1.0818\mathcal L=-\log0.339=1.0818. Check by brute force: the 5 alignments are aab (0.0600), abb (0.0800), -ab (0.0750), a-b (0.0600) and ab- (0.0640), which also sum to 0.3390. The skip into s=4s=4 at t=2t=2 (from the label aa straight to bb) is the term that makes ab adjacency possible.

tβ_t (s=1…5)Σγ for ε, a, b∂L/∂u = y − Σγ
10.0750, 0.2640, 0.2550, 0.0480, 0.06000.2212, 0.7788, 0(+0.2788, −0.3788, +0.1000)
20, 0.1500, 0.1500, 0.3600, 0.12000.1770, 0.3982, 0.4248(+0.1230, −0.0982, −0.0248)
30, 0, 0, 0.5000, 0.40000.1888, 0, 0.8112(+0.2112, +0.1000, −0.3112)
Posterior occupancy and gradient per frame

At t=1t=1 the occupancy is γ1(2)=α1(2)β1(2)/(y1(a) p)=0.4⋅0.264/(0.4⋅0.339)=0.7788\gamma_1(2)=\alpha_1(2)\beta_1(2)/(y_1(a)\,p)=0.4\cdot0.264/(0.4\cdot0.339)=0.7788: 78% of the alignment mass has "a" in frame 1. The network gives "a" only 0.4, so its logit is pushed up (−0.3788-0.3788), and blank and "b" are pushed down. At t=2t=2, y2(b)=0.4y_2(b)=0.4 almost matches its posterior 0.4248, so bb gets only a small push (−0.0248-0.0248). Every row of the gradient sums to 0.

Log space, not rescaling

α is a product of TT probabilities, so it underflows quickly. Graves (2006) rescaled α and β at each frame by their sum and accumulated the log of the scale factors. Modern implementations (PyTorch, warp-ctc, cuDNN, Optax) instead store log⁡α\log\alpha and replace every sum by log-sum-exp, log⁡(ea+eb)=m+log⁡(ea−m+eb−m)\log(e^a+e^b)=m+\log(e^{a-m}+e^{b-m}) with m=max⁡(a,b)m=\max(a,b), and log⁡0=−∞\log0=-\infty for unreachable states. Inputs are log-probabilities from log_softmax, so the network output never passes through a raw probability.

In PyTorch: nn.CTCLoss

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3ctc = nn.CTCLoss(blank=0, reduction="mean", zero_infinity=True)
4
5# a batch: T frames (after any downsampling!), N utterances, C = labels + 1 blank
6T, N, C = 50, 4, 28
7logits = torch.randn(T, N, C, requires_grad=True) # time-major: (T, N, C)
8log_probs = logits.log_softmax(dim=-1) # log-probs, NOT softmax probs
9target_lengths = torch.tensor([12, 7, 20, 3])
10targets = torch.randint(1, C, (N, int(target_lengths.max()))) # padded (N, S); never use 0 = blank
11input_lengths = torch.tensor([50, 41, 50, 18]) # valid frames per utterance, each <= T
12loss = ctc(log_probs, targets, input_lengths, target_lengths)
13loss.backward()
14
15# the worked example: T=3, alphabet (blank, a, b), target "ab"
16y = torch.tensor([[0.5, 0.4, 0.1], [0.3, 0.3, 0.4], [0.4, 0.1, 0.5]])
17u = y.log().unsqueeze(1).requires_grad_() # (T=3, N=1, C=3) logits
18lp = u.log_softmax(-1)
19nll = F.ctc_loss(lp, torch.tensor([[1, 2]]), torch.tensor([3]), torch.tensor([2]), reduction="sum")
20nll.backward()
21print(nll) # 1.0818 = -log 0.339
22print(u.grad.squeeze(1)) # [[ 0.2788, -0.3788, 0.1000],
23 # [ 0.1230, -0.0982, -0.0248],
24 # [ 0.2112, 0.1000, -0.3112]]
Shapes, a padded batch, and the worked example reproduced
ArgumentShape / valueMeaning
log_probs(T, N, C) or (T, C)log-softmax outputs, time first; float32 is safest
targets(N, S) padded, or 1-D of length Σ target_lengthslabel indices in 1…C−1 (anything except blank)
input_lengths(N,)valid frames per sequence, ≤ T; padding frames beyond it are ignored
target_lengths(N,)true length of each target; padding in targets beyond it is ignored
blankint, default 0index of the blank in the C classes
reduction'mean' (default), 'sum', 'none'mean divides each loss by its target length, then averages over the batch; none returns (N,)
zero_infinitybool, default Falsereplace infinite losses (infeasible T) and their gradients by 0
nn.CTCLoss arguments

Use when

  • +Exact marginal likelihood over all alignments in O(T·U), no frame labels required.
  • +Simple, bounded gradient: softmax minus posterior occupancy, like cross-entropy with soft targets.
  • +Fully supported and fast in every framework (PyTorch native and cuDNN, TensorFlow, Optax).
  • +Works as an auxiliary loss on top of attention or transducer models to enforce monotonic alignment.

Avoid when

  • −Infinite for infeasible lengths; one bad sample can NaN a whole batch unless lengths are checked.
  • −Sequential over frames: the α/β recursion cannot be parallelised over time.
  • −The loss value depends on reduction and length normalisation, which makes comparisons across setups tricky.
  • −Inherits CTC's conditional-independence assumption: the loss itself has no notion of label context.

Check yourself

For the target "abb", which skip transition is forbidden in the extended sequence l′=(ε,a,ε,b,ε,b,ε)l'=(\varepsilon,a,\varepsilon,b,\varepsilon,b,\varepsilon)?

Check yourself

At some frame the network predicts yt(ε)=0.7y_t(\varepsilon)=0.7, and the alignment posterior puts total occupancy 0.90.9 on blank states. What is ∂L/∂ut(ε)\partial\mathcal L/\partial u_t(\varepsilon), and which way does gradient descent move the blank logit?

Remember

  • ✓Pad the target with blanks: l′l' has 2U+12U+1 states; moves are stay, advance, or skip a blank between different labels.
  • ✓αt(s)=(αt−1(s)+αt−1(s−1)+[skip]αt−1(s−2)) yt(ls′)\alpha_t(s)=(\alpha_{t-1}(s)+\alpha_{t-1}(s-1)+[\text{skip}]\alpha_{t-1}(s-2))\,y_t(l'_s); p(y∣x)=αT(S)+αT(S−1)p(y\mid x)=\alpha_T(S)+\alpha_T(S-1).
  • ✓∑sαt(s)βt(s)/yt(ls′)=p(y∣x)\sum_s\alpha_t(s)\beta_t(s)/y_t(l'_s)=p(y\mid x) at every tt: a built-in correctness check.
  • ✓Gradient w.r.t. logits: yt(k)−∑s:ls′=kγt(s)y_t(k)-\sum_{s:l'_s=k}\gamma_t(s), cross-entropy against the model's own soft alignment.
  • ✓Compute in log space with log-sum-exp; probability-space α underflows within a few hundred frames.
  • ✓nn.CTCLoss: time-major log-probs (T, N, C), blank never in targets, input_lengths after subsampling, 'mean' divides by target length.
  • ✓Inf loss means T<U+rT<U+r for some sample: fix the lengths, use zero_infinity only as a guard.
← back to Deep Learning roadmap