# CTC loss

> −log p(y|x) summed over every alignment by a forward–backward pass over the blank-padded target (2U+1 states). The gradient on each logit is softmax minus posterior occupancy; in PyTorch it is nn.CTCLoss on (T, N, C) log-probs.

Source: https://www.mlfoundry.in/learn/ctc-loss  
Section: Deep Learning › Loss Functions  
Level: advanced · 14 min read · updated 2026-10-11

**CTC loss:** is the negative log-likelihood $\mathcal L=-\log p(y\mid x)$ of a target label sequence $y$ under **Connectionist Temporal Classification**, where $p(y\mid x)$ is the total probability of every frame-level path that collapses to $y$. It is computed exactly, in $O(T\cdot U)$ time, by a forward–backward dynamic programme over the target with blanks inserted.

**Connectionist Temporal Classification** explains the model: a per-frame softmax over labels plus a blank $\varepsilon$, the collapse map $\mathcal B$ (merge repeats, then drop blanks), and $p(y\mid x)=\sum_{\pi\in\mathcal B^{-1}(y)}\prod_t y_t(\pi_t)$. This article is about computing that sum and its gradient, and about using `nn.CTCLoss` without the usual mistakes. The sum has $\binom{T+U}{2U}$ terms for a repeat-free target (about $1.4\times10^9$ for 3 labels in 100 frames), so it cannot be enumerated. The recursion below is from Graves, Fernández, Gomez and Schmidhuber (ICML 2006). It is the same forward–backward idea as for hidden Markov models.

## The extended label sequence

Insert a blank at the start, between every pair of labels and at the end: $y=(a,b)$ becomes $l'=(\varepsilon,a,\varepsilon,b,\varepsilon)$. A target of $U$ labels has $S=2U+1$ states. Every alignment is a monotone walk through $l'$ that spends at least one frame on each label and may skip any blank, except one that sits between two equal labels.

**Allowed moves from frame t−1 to frame t**

1. **Stay** in state $s$: a label or blank held for another frame (merged by $\mathcal B$).
2. **Advance** from $s-1$ to $s$: blank → label or label → blank.
3. **Skip** from $s-2$ to $s$ over a blank, allowed only if $l'_s\ne\varepsilon$ and $l'_s\ne l'_{s-2}$. Two different labels may touch (`ab`); two equal ones may not, or `aa` would merge into one `a`.
4. **Start** in state 1 (leading blank) or state 2 (first label). **End** in state $S$ (trailing blank) or $S-1$ (last label).

## Forward (α) and backward (β) recursions

$$
\alpha_t(s)=\Big(\alpha_{t-1}(s)+\alpha_{t-1}(s-1)+\mathbb 1[\text{skip}]\,\alpha_{t-1}(s-2)\Big)\,y_t(l'_s),\qquad \alpha_1(1)=y_1(\varepsilon),\;\;\alpha_1(2)=y_1(l'_2)
$$

- $\alpha_t(s)$: total probability of all path prefixes $\pi_{1:t}$ that are valid starts of an alignment and sit in state $s$ at frame $t$
- $y_t(k)$: softmax output for symbol $k$ at frame $t$
- $\mathbb 1[\text{skip}]$: 1 if $l'_s\ne\varepsilon$ and $l'_s\ne l'_{s-2}$, else 0; all other $\alpha_1(s)=0$

$$
p(y\mid x)=\alpha_T(S)+\alpha_T(S-1),\qquad \mathcal L_{\text{CTC}}=-\log p(y\mid x)
$$

- $S$: $2U+1$, the trailing blank; $S-1$ is the last label

The backward variable runs the same recursion from the end, $\beta_t(s)=\big(\beta_{t+1}(s)+\beta_{t+1}(s+1)+\mathbb 1[\text{skip into }s+2]\,\beta_{t+1}(s+2)\big)\,y_t(l'_s)$ with $\beta_T(S)=y_T(\varepsilon)$, $\beta_T(S-1)=y_T(l'_{S-1})$. With this convention both $\alpha_t(s)$ and $\beta_t(s)$ include the frame-$t$ output, so their product counts it twice. Dividing once gives the total mass of all complete alignments that pass through $(t,s)$:

$$
\frac{\alpha_t(s)\,\beta_t(s)}{y_t(l'_s)}=\sum_{\pi\in\mathcal B^{-1}(y):\;\pi\text{ in state }s\text{ at }t} p(\pi\mid x),\qquad \sum_{s=1}^{S}\frac{\alpha_t(s)\beta_t(s)}{y_t(l'_s)}=p(y\mid x)\;\;\text{for every }t
$$

- $\gamma_t(s)$: $\alpha_t(s)\beta_t(s)/\big(y_t(l'_s)\,p(y\mid x)\big)$, the posterior probability that frame $t$ is aligned to state $s$; $\sum_s\gamma_t(s)=1$

## Gradient with respect to the logits

Let $u_t(k)$ be the pre-softmax logits, $y_t=\operatorname{softmax}(u_t)$. Differentiating $-\log p$ through the recursion and the softmax (Graves et al., 2006, eq. 16) gives:

$$
\frac{\partial\mathcal L}{\partial u_t(k)}=y_t(k)-\frac{1}{p(y\mid x)}\sum_{s:\,l'_s=k}\frac{\alpha_t(s)\beta_t(s)}{y_t(k)}=y_t(k)-\sum_{s:\,l'_s=k}\gamma_t(s)
$$

- $\sum_{s:l'_s=k}\gamma_t(s)$: posterior probability that frame $t$ emits symbol $k$, summed over every state carrying $k$ (all blank states for $k=\varepsilon$)
- $y_t(k)$: what the network currently predicts for frame t

This is the **softmax cross-entropy gradient $p-q$ with soft frame targets** $q_t(k)=\sum_{s:l'_s=k}\gamma_t(s)$. The targets are not given; they are the model's own posterior over alignments, recomputed every step, which makes CTC training an EM-like loop: infer a soft alignment (E), fit the frame classifier to it (M). Each component lies in $[-1,1]$ and each frame's gradient sums to 0.

**At a glance**

- **Loss:** $-\log p(y\mid x)\ge0$; 0 only if one alignment has probability 1; $+\infty$ if no alignment fits in $T$ frames
- **States:** $S=2U+1$ (blank-padded target)
- **Cost:** $O(T\cdot S)$ time and memory for α and β, per utterance; frames are sequential, the batch and states parallelise
- **Feasibility:** $T\ge U+r$, $r$ = number of adjacent equal labels (each needs a separating blank)
- **Gradient:** $y_t-q_t$ per frame: softmax minus posterior occupancy, bounded, sums to 0
- **Numerics:** run in log space with log-sum-exp; in probability space α underflows float32 after ~126 frames at $y\approx0.5$ ($0.5^{126}\approx1.2\times10^{-38}$)
- **PyTorch:** `nn.CTCLoss(blank=0, reduction="mean", zero_infinity=False)`, inputs `(log_probs, targets, input_lengths, target_lengths)`

## Interactive: the α lattice

> Figure: Rows are the 2U+1 states of the blank-extended target, columns are frames, and the frame posteriors come from fixed random logits. Default: target "abc", T = 6, so 7 states and T ≥ 3. p(y|x) = α₆(7) + α₆(6) = 0.0537 and the loss is 2.9252. Click a cell to see its recursion: α₄(4) for the label b is (α₃(4) + α₃(3) + α₃(2)) × y₄(b) = (0.1112 + 0.0264 + 0.1620) × 0.514 ≈ 0.154. The skip from α₃(2) is allowed because a ≠ b. Grey cells lie on no complete path. Violet edges are skips; choose "abb" and the skip into the second b disappears. Drop T to 2 and every target becomes infeasible (loss = inf). Switch to β or γ and check that Σₛ αβ/y equals p at every frame. Train with free logits: the loss falls to 0.4309 after 10 steps and 0.0402 after 100, and γ concentrates on one alignment (argmax path εaabbc).

## Worked example

Alphabet $\{\varepsilon,a,b\}$, target $y=(a,b)$, $T=3$, so $l'=(\varepsilon,a,\varepsilon,b,\varepsilon)$. The frame posteriors are $y_1=(0.5,0.4,0.1)$, $y_2=(0.3,0.3,0.4)$, $y_3=(0.4,0.1,0.5)$ over $(\varepsilon,a,b)$.

α_t(s): forward lattice (exact to 4 decimals)

| t | s=1 ε | s=2 a | s=3 ε | s=4 b | s=5 ε |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.5000 | 0.4000 | 0 | 0 | 0 |
| 2 | $0.5\cdot0.3=0.1500$ | $(0.4+0.5)\cdot0.3=0.2700$ | $(0+0.4)\cdot0.3=0.1200$ | $(0+0+0.4)\cdot0.4=0.1600$ | 0 |
| 3 | 0.0600 | 0.0420 | 0.1560 | $(0.16+0.12+0.27)\cdot0.5=0.2750$ | $(0+0.16)\cdot0.4=0.0640$ |

So $p(y\mid x)=\alpha_3(5)+\alpha_3(4)=0.0640+0.2750=0.3390$ and $\mathcal L=-\log0.339=1.0818$. Check by brute force: the 5 alignments are `aab` (0.0600), `abb` (0.0800), `-ab` (0.0750), `a-b` (0.0600) and `ab-` (0.0640), which also sum to 0.3390. The skip into $s=4$ at $t=2$ (from the label $a$ straight to $b$) is the term that makes `ab` adjacency possible.

Posterior occupancy and gradient per frame

| t | β_t (s=1…5) | Σγ for ε, a, b | ∂L/∂u = y − Σγ |
| --- | --- | --- | --- |
| 1 | 0.0750, 0.2640, 0.2550, 0.0480, 0.0600 | 0.2212, 0.7788, 0 | (+0.2788, −0.3788, +0.1000) |
| 2 | 0, 0.1500, 0.1500, 0.3600, 0.1200 | 0.1770, 0.3982, 0.4248 | (+0.1230, −0.0982, −0.0248) |
| 3 | 0, 0, 0, 0.5000, 0.4000 | 0.1888, 0, 0.8112 | (+0.2112, +0.1000, −0.3112) |

At $t=1$ the occupancy is $\gamma_1(2)=\alpha_1(2)\beta_1(2)/(y_1(a)\,p)=0.4\cdot0.264/(0.4\cdot0.339)=0.7788$: 78% of the alignment mass has "a" in frame 1. The network gives "a" only 0.4, so its logit is pushed up ($-0.3788$), and blank and "b" are pushed down. At $t=2$, $y_2(b)=0.4$ almost matches its posterior 0.4248, so $b$ gets only a small push ($-0.0248$). Every row of the gradient sums to 0.

## Log space, not rescaling

α is a product of $T$ probabilities, so it underflows quickly. Graves (2006) rescaled α and β at each frame by their sum and accumulated the log of the scale factors. Modern implementations (PyTorch, warp-ctc, cuDNN, Optax) instead store $\log\alpha$ and replace every sum by log-sum-exp, $\log(e^a+e^b)=m+\log(e^{a-m}+e^{b-m})$ with $m=\max(a,b)$, and $\log0=-\infty$ for unreachable states. Inputs are log-probabilities from `log_softmax`, so the network output never passes through a raw probability.

> **Only a band of the lattice is live:** A state can be reached at frame $t$ only if enough frames have passed to emit the labels before it, and left only if enough remain for the labels after it. Cells outside this band have $\alpha\beta=0$ (grey in the figure). Efficient kernels exploit this, and it is exactly why a too-short $T$ leaves no live path at all.

## In PyTorch: nn.CTCLoss

```python
import torch, torch.nn as nn, torch.nn.functional as F

ctc = nn.CTCLoss(blank=0, reduction="mean", zero_infinity=True)

# a batch: T frames (after any downsampling!), N utterances, C = labels + 1 blank
T, N, C = 50, 4, 28
logits = torch.randn(T, N, C, requires_grad=True)     # time-major: (T, N, C)
log_probs = logits.log_softmax(dim=-1)                # log-probs, NOT softmax probs
target_lengths = torch.tensor([12, 7, 20, 3])
targets = torch.randint(1, C, (N, int(target_lengths.max())))  # padded (N, S); never use 0 = blank
input_lengths = torch.tensor([50, 41, 50, 18])        # valid frames per utterance, each <= T
loss = ctc(log_probs, targets, input_lengths, target_lengths)
loss.backward()

# the worked example: T=3, alphabet (blank, a, b), target "ab"
y = torch.tensor([[0.5, 0.4, 0.1], [0.3, 0.3, 0.4], [0.4, 0.1, 0.5]])
u = y.log().unsqueeze(1).requires_grad_()            # (T=3, N=1, C=3) logits
lp = u.log_softmax(-1)
nll = F.ctc_loss(lp, torch.tensor([[1, 2]]), torch.tensor([3]), torch.tensor([2]), reduction="sum")
nll.backward()
print(nll)                # 1.0818  = -log 0.339
print(u.grad.squeeze(1))  # [[ 0.2788, -0.3788,  0.1000],
                          #  [ 0.1230, -0.0982, -0.0248],
                          #  [ 0.2112,  0.1000, -0.3112]]
```

Shapes, a padded batch, and the worked example reproduced

nn.CTCLoss arguments

| Argument | Shape / value | Meaning |
| --- | --- | --- |
| `log_probs` | (T, N, C) or (T, C) | log-softmax outputs, **time first**; float32 is safest |
| `targets` | (N, S) padded, or 1-D of length Σ target_lengths | label indices in 1…C−1 (anything except blank) |
| `input_lengths` | (N,) | valid frames per sequence, ≤ T; padding frames beyond it are ignored |
| `target_lengths` | (N,) | true length of each target; padding in `targets` beyond it is ignored |
| `blank` | int, default 0 | index of the blank in the C classes |
| `reduction` | `'mean'` (default), `'sum'`, `'none'` | `mean` divides each loss by its target length, then averages over the batch; `none` returns (N,) |
| `zero_infinity` | bool, default False | replace infinite losses (infeasible T) and their gradients by 0 |

> **Why is my CTC loss inf (or NaN)?:** (1) **Infeasible lengths**: some `input_lengths[i] < target_lengths[i] + repeats`, very often because `input_lengths` were taken from the raw audio and not divided by the encoder's subsampling factor (or computed before it). Fix the lengths; `zero_infinity=True` only hides the bad samples. (2) **Batch-first input**: `(N, T, C)` passed as `(T, N, C)` gives garbage or inf without an error when shapes happen to fit; transpose with `.transpose(0, 1)`. (3) **Probabilities instead of log-probs**: passing `softmax` output silently computes the wrong loss. (4) **Blank used as a label**: a real character mapped to index 0. (5) **Half precision**: compute `log_softmax` and the loss in float32 under AMP.

> **'mean' is not the mean over frames:** With `reduction="mean"` each utterance's $-\log p$ is divided by its **target** length before averaging over the batch, so the value is "nats per label", not per sequence or per frame. Switching to `"sum"` multiplies the gradient scale by roughly $N\cdot\bar U$, so the learning rate has to change with it.

> **The gradient explains peaky outputs:** Blank appears in $U+1$ of the $2U+1$ states and is allowed on every frame of every utterance, so early in training its posterior occupancy $q_t(\varepsilon)$ is large almost everywhere and the gradient $y_t-q_t$ pushes blank up across the board. Labels only win the frames where the alignment posterior concentrates them. The result is the blank-dominated, spiky output described in **Connectionist Temporal Classification**. With free per-frame logits, as in the figure, training simply concentrates γ on one alignment.

**Strengths**

- Exact marginal likelihood over all alignments in O(T·U), no frame labels required.
- Simple, bounded gradient: softmax minus posterior occupancy, like cross-entropy with soft targets.
- Fully supported and fast in every framework (PyTorch native and cuDNN, TensorFlow, Optax).
- Works as an auxiliary loss on top of attention or transducer models to enforce monotonic alignment.

**Limitations**

- Infinite for infeasible lengths; one bad sample can NaN a whole batch unless lengths are checked.
- Sequential over frames: the α/β recursion cannot be parallelised over time.
- The loss value depends on reduction and length normalisation, which makes comparisons across setups tricky.
- Inherits CTC's conditional-independence assumption: the loss itself has no notion of label context.

**Question:** For the target "abb", which skip transition is **forbidden** in the extended sequence $l'=(\varepsilon,a,\varepsilon,b,\varepsilon,b,\varepsilon)$?

- A. from $a$ (s=2) to the first $b$ (s=4)
- B. from the first $b$ (s=4) to the second $b$ (s=6)
- C. from $\varepsilon$ (s=1) to $a$ (s=2)
- D. none: every skip over a blank is allowed

**Answer:** B. from the first $b$ (s=4) to the second $b$ (s=6) — A skip $s-2\to s$ needs $l'_s\ne l'_{s-2}$. Going straight from $b$ to $b$ would put `bb` on adjacent frames, which $\mathcal B$ merges into one $b$, so the path must pass through the blank at s=5. Option 3 is an ordinary advance, not a skip.

**Question:** At some frame the network predicts $y_t(\varepsilon)=0.7$, and the alignment posterior puts total occupancy $0.9$ on blank states. What is $\partial\mathcal L/\partial u_t(\varepsilon)$, and which way does gradient descent move the blank logit?

- A. +0.2, down
- B. −0.2, up
- C. −0.9, up
- D. +0.7, down

**Answer:** B. −0.2, up — $\partial\mathcal L/\partial u_t(k)=y_t(k)-q_t(k)=0.7-0.9=-0.2$. Descent subtracts the gradient, so the blank logit goes up until the prediction matches the posterior.

**Key points**

- Pad the target with blanks: $l'$ has $2U+1$ states; moves are stay, advance, or skip a blank between **different** labels.
- $\alpha_t(s)=(\alpha_{t-1}(s)+\alpha_{t-1}(s-1)+[\text{skip}]\alpha_{t-1}(s-2))\,y_t(l'_s)$; $p(y\mid x)=\alpha_T(S)+\alpha_T(S-1)$.
- $\sum_s\alpha_t(s)\beta_t(s)/y_t(l'_s)=p(y\mid x)$ at every $t$: a built-in correctness check.
- Gradient w.r.t. logits: $y_t(k)-\sum_{s:l'_s=k}\gamma_t(s)$, cross-entropy against the model's own soft alignment.
- Compute in log space with log-sum-exp; probability-space α underflows within a few hundred frames.
- `nn.CTCLoss`: time-major log-probs (T, N, C), blank never in targets, `input_lengths` after subsampling, `'mean'` divides by target length.
- Inf loss means $T<U+r$ for some sample: fix the lengths, use `zero_infinity` only as a guard.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
