# Connectionist Temporal Classification

> Label unsegmented sequences by adding a blank symbol and summing over every frame-level alignment that collapses to the target. No frame labels are needed, but outputs are conditionally independent and peaky.

Source: https://www.mlfoundry.in/learn/connectionist-temporal-classification  
Section: Deep Learning › Loss Functions  
Level: advanced · 13 min read · updated 2026-10-11

**Connectionist Temporal Classification (CTC):** is a way to train and decode a network that reads a long input sequence $x_{1:T}$ (audio frames, image columns, pen strokes) and must output a shorter label sequence $y_{1:U}$, $U\le T$, **without knowing which frames belong to which label**. The network emits a distribution over the labels plus a special **blank** $\varepsilon$ at every frame; a frame-level path is mapped to a label sequence by merging repeats and deleting blanks, and $p(y\mid x)$ is the total probability of every path that maps to $y$.

Speech is the classic case. One second of audio is 100 feature frames at a 10 ms hop, but the transcript "cat" is three characters, and nobody has marked where the "c" ends and the "a" begins. Before CTC, recognisers needed that segmentation: an HMM produced frame-level targets by forced alignment, and the neural network was trained as a frame classifier on them. CTC removes the alignment step: the network is trained directly on (input, transcript) pairs.

It was introduced by Alex Graves, Santiago Fernández, Faustino Gomez and Jürgen Schmidhuber at ICML 2006 ("Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks"), with bidirectional LSTMs on TIMIT phoneme recognition. Graves and colleagues then used it for unconstrained handwriting recognition (2009), Baidu's Deep Speech (Hannun et al., 2014) built an end-to-end speech recogniser on it, and the CRNN of Shi, Bai and Yao (2015) made it the standard loss for scene-text OCR. Self-supervised speech encoders such as wav2vec 2.0 (Baevski et al., 2020) are still usually fine-tuned with a CTC head.

This article covers the model: the blank, the collapse map, the alignment set, the assumptions and decoding. How $-\log p(y\mid x)$ and its gradient are actually computed (the forward–backward recursion, PyTorch's `nn.CTCLoss`) is the subject of **CTC loss**.

## The blank and the collapse map B

Let $L$ be the label alphabet and $L'=L\cup\{\varepsilon\}$. At each frame the network outputs a softmax $y_t(k)=p(k\mid x,t)$ over $L'$. A **path** (or alignment) is a sequence $\pi\in L'^{\,T}$ with one symbol per frame. The many-to-one map $\mathcal B:L'^{\,T}\to L^{\le T}$ turns a path into a labelling in two steps:

**B(π): collapse a path**

1. **Merge consecutive repeats**: `CC-AA--T` becomes `C-A-T`. A label held for several frames counts once.
2. **Delete every blank**: `C-A-T` becomes `CAT`.
3. The order matters. `CA-AT` merges nothing (the two A's are separated by a blank) and gives `CAAT`; `CAAT` as a path merges to `CAT`. A blank between two equal labels is the **only** way to emit a doubled letter.

The blank does two jobs. It lets the network say "nothing new here" on the many frames between labels (silence, transitions, the inside of a long vowel), and it separates genuine repeats such as the "ll" in "hello". Without it, the network could never output "aa" at all.

## The probability of a labelling

$$
p(y\mid x)\;=\;\sum_{\pi\in\mathcal B^{-1}(y)} p(\pi\mid x),\qquad p(\pi\mid x)\;=\;\prod_{t=1}^{T} y_t(\pi_t)
$$

- $y_t(k)$: network softmax output for symbol $k\in L'$ at frame $t$
- $\pi$: a path: one symbol of $L'$ per frame, $|\pi|=T$
- $\mathcal B^{-1}(y)$: the set of all paths that collapse to the target $y$, its **alignments**
- $\prod_t$: conditional independence: given $x$, the frame outputs are independent

Training maximises $\log p(y\mid x)$, i.e. minimises $-\log p(y\mid x)$. The set of alignments is large: for a target of $U$ labels with no adjacent repeats there are exactly $\binom{T+U}{2U}$ of them (a path is $U$ label runs of length $\ge1$ interleaved with $U+1$ blank runs of length $\ge0$). "CAT" has exactly 1 alignment in 3 frames (`CAT` itself) and 7 in 4 frames; in 8 frames it has $\binom{11}{6}=462$; at $T=100$ frames there are about $1.4\times10^{9}$. The sum is never enumerated: a dynamic programme over the blank-extended target computes it in $O(T\cdot U)$ (see **CTC loss**).

**At a glance**

- **Task:** sequence labelling with $U\le T$ and monotonic alignment: speech recognition, handwriting, OCR, sign language, keyword spotting
- **Output layer:** per-frame softmax over $|L|+1$ symbols (labels plus blank); blank is index 0 by convention
- **Supervision:** only the label sequence; no frame boundaries
- **Length constraint:** $T\ge U+r$, where $r$ is the number of adjacent equal labels in $y$ (each needs a blank between)
- **Alignments:** $\binom{T+U}{2U}$ for $U$ labels without repeats; fewer with repeats (e.g. 165 for "CAAT" vs 462 for "CAT" at $T=8$)
- **Assumption:** frame outputs independent given $x$; no model of label-to-label dependencies inside the loss
- **Decoding:** greedy best path (fast, approximate), prefix beam search, beam search with an external language model

## Interactive: alignments, collapse and decoding

> Figure: An 8-frame posteriorgram over {ε, C, A, T}. The outlined path starts as the greedy argmax; click cells to edit it and watch B merge repeats and drop blanks. On the default 'ambiguous' network greedy decoding reads C----T-- → "CT" (p(CT|x) = 0.0568), but summing all 65 536 paths shows "CAT" is far more likely (0.2460): its A is spread over three frames where blank narrowly wins each, and no single CAT alignment carries more than 4.8% of its mass. Prefix beam search with width 1 also returns CT; width 2 already finds CAT. On the 'clean' network greedy is right and one alignment, -C-A--T-, carries 65.4% of p(CAT|x) = 0.4752. τ < 1 sharpens every frame, τ > 1 flattens.

## Worked example: two frames

Take $T=2$ and a one-letter alphabet $\{\varepsilon,a\}$, with $y_t(\varepsilon)=0.6$, $y_t(a)=0.4$ at both frames. Every path and where $\mathcal B$ sends it:

All 4 paths for T = 2

| Path π | B(π) | p(π\|x) |
| --- | --- | --- |
| `--` | ∅ (empty) | $0.6\times0.6=0.36$ |
| `-a` | a | $0.6\times0.4=0.24$ |
| `a-` | a | $0.4\times0.6=0.24$ |
| `aa` | a | $0.4\times0.4=0.16$ |

The single most probable path is `--` (0.36), so greedy decoding outputs the empty string. But $p(a\mid x)=0.24+0.24+0.16=0.64$: the labelling "a" is almost twice as likely, because its mass is split over three alignments. "aa" is impossible here ($p=0$), since it needs at least 3 frames (`a-a`). This gap between the most probable **path** and the most probable **labelling** is what beam search exists to close.

## Assumptions and their consequences

**Monotonic alignment.** Paths move left to right through the target and can never go back. That is right for speech and handwriting, wrong for translation, where word order changes.

**Conditional independence.** $p(\pi\mid x)=\prod_t y_t(\pi_t)$: once the encoder has seen the whole input, each frame's choice ignores the choices at other frames. A bidirectional or Transformer encoder can still use plenty of context to produce $y_t$, but the loss cannot express "after 'q' comes 'u'". Character CTC models therefore make phonetically plausible spelling errors, and an external language model helps a lot.

**Length.** Each label needs at least one frame, and each adjacent repeat needs an extra blank frame: $T\ge U+r$. Encoders that downsample (convolutional front ends reduce frame rate by 4–8× in modern ASR) can violate this for fast speech or long transcripts, and the loss becomes infinite (see **CTC loss** for `zero_infinity`).

> **Peaky behaviour:** Trained CTC networks are famously **peaky**: almost every frame puts nearly all its probability on blank, and each label appears as a sharp one- or two-frame spike, often not at the acoustic centre of the sound (Graves et al. noted the spiky outputs in 2006; Zeyer, Schlüter and Ney, 2021, analyse why). Blank is a safe bet on every frame of every utterance, so it is the easiest symbol to learn and quickly dominates, and once the target is fit, the loss gains nothing from spreading a label over more frames. Consequences: greedy decoding works surprisingly well on well-trained models, but CTC spike positions are a poor source of timestamps or phone boundaries.

## Decoding

The ideal output is $\hat y=\arg\max_y p(y\mid x)$, a search over labellings in which each candidate is itself a sum over alignments. It is intractable in general, so three approximations are used.

- **Greedy (best path):** Take $\arg\max_k y_t(k)$ at every frame, then apply $\mathcal B$. $O(T|L'|)$, no search. It finds the most probable **path**, which is not necessarily the most probable labelling (the two-frame example, and the 'ambiguous' preset in the figure). On peaky, well-trained models the dominant path usually carries most of the mass and the two agree.
- **Prefix beam search:** Keep the $W$ best label **prefixes**, not paths. Each prefix stores two numbers, the probability of all paths ending in blank, $p_b$, and ending in its last label, $p_{nb}$, so that at the next frame a repeated label can be merged (extends from $p_{nb}$) or appended as a new label (only from $p_b$). Paths that collapse to the same prefix are summed instead of competing.
- **Beam search with an LM (shallow fusion):** Score prefixes by $\log p_{\text{CTC}}(\ell\mid x)+\alpha\log p_{\text{LM}}(\ell)+\beta|\ell|$, where the LM is an n-gram or neural model over characters or words, $\alpha$ is its weight and $\beta$ a length (word-insertion) bonus that offsets the LM's preference for short outputs. Deep Speech used exactly this form. WFST decoders (as in EESEN and Kaldi-style pipelines) compile lexicon and grammar into the search graph.

> **Greedy decoding needs B, not just argmax:** A common bug is to take the per-frame argmax and then only delete blanks, or only merge repeats. Both steps are required, and in that order: `C-AAT` must become `CAT`, and `A-A` must stay `AA`. Also remove the blank index **after** merging, never before: deleting blanks first turns `A-A` into `AA` and then merges it to `A`.

```python
import torch
from collections import defaultdict

def greedy_decode(log_probs, blank=0):          # log_probs: (T, C) for one utterance
    best = log_probs.argmax(dim=-1).tolist()      # best path, one symbol per frame
    out, prev = [], None
    for k in best:                                # B: merge repeats, then drop blanks
        if k != prev and k != blank:
            out.append(k)
        prev = k
    return out

def prefix_beam_search(probs, width=8, blank=0):  # probs: (T, C) softmax outputs
    beam = {(): (1.0, 0.0)}                       # prefix -> (p_blank_end, p_label_end)
    for row in probs.tolist():
        nxt = defaultdict(lambda: [0.0, 0.0])
        for prefix, (pb, pnb) in beam.items():
            nxt[prefix][0] += (pb + pnb) * row[blank]           # emit blank
            for k, pk in enumerate(row):
                if k == blank:
                    continue
                if prefix and prefix[-1] == k:
                    nxt[prefix][1] += pnb * pk                   # repeat: merged
                    nxt[prefix + (k,)][1] += pb * pk             # after a blank: new label
                else:
                    nxt[prefix + (k,)][1] += (pb + pnb) * pk
        beam = dict(sorted(nxt.items(), key=lambda kv: -sum(kv[1]))[:width])
    return max(beam.items(), key=lambda kv: sum(kv[1]))

# the two-frame example: blank 0.6, "a" 0.4 at both frames
probs = torch.tensor([[0.6, 0.4], [0.6, 0.4]])
print(greedy_decode(probs.log()))        # []        best path "--"
best, (pb, pnb) = prefix_beam_search(probs)
print(best, pb + pnb)                    # (1,) ≈0.64 -> "a": 0.24 ending in blank + 0.40 ending in "a"
```

Greedy CTC decoding and a minimal prefix beam search (blank = 0, no language model)

In production use a tested implementation: `torchaudio.models.decoder.ctc_decoder` (a lexicon- and KenLM-aware beam search from Flashlight) or `pyctcdecode`. Probabilities in real models should be handled in log space; the toy above multiplies raw probabilities for readability.

## CTC versus attention and transducers

Three ways to train a sequence-to-sequence recogniser

| Aspect | CTC | Attention encoder–decoder | RNN-Transducer |
| --- | --- | --- | --- |
| Alignment | monotonic, marginalised | learned soft attention, any order | monotonic, marginalised |
| Label dependencies | none (conditionally independent) | full, autoregressive decoder | yes, via a prediction network |
| Output length | $U\le T$ required | unrestricted | unrestricted (emits $\ge0$ labels per frame) |
| Inference | one parallel pass + collapse | sequential, beam search | sequential over frames and labels |
| Streaming | natural | hard (attends to the whole input) | natural |
| Typical failure | spelling errors without an LM | looping, skipping, hallucinated words on long inputs | memory-hungry loss over a T×U×V lattice |

The attention route is Listen, Attend and Spell (Chan et al., 2016); the transducer is Graves (2012), "Sequence Transduction with Recurrent Neural Networks", which removes CTC's independence assumption by conditioning each output on previous labels. In practice they are often combined: joint CTC–attention training (Kim, Hori and Watanabe, 2017) uses a CTC head as an auxiliary loss that forces the attention to stay monotonic, and decoding mixes both scores.

**Strengths**

- Trains on transcripts alone: no frame-level segmentation or forced alignment needed.
- Non-autoregressive: one forward pass gives every frame's output, so greedy decoding is fast and trivially parallel.
- Monotonic alignment is a strong, correct prior for speech, handwriting and OCR; it cannot loop or skip like attention can.
- Streams naturally with a causal or chunked encoder.
- Plays well with external language models and lexicons at decoding time.

**Limitations**

- Conditional independence: no output-side language model inside the loss, so accuracy depends on the encoder and an external LM.
- Needs $T\ge U+r$: unusable when the output can be longer than the input, or for non-monotonic tasks such as translation.
- Peaky, blank-dominated outputs give unreliable timing and phone boundaries.
- Most probable path ≠ most probable labelling; exact decoding is intractable.

**Question:** Which path collapses to "HELLO" under $\mathcal B$?

- A. `HHEELLLOO`
- B. `HE-LLO`
- C. `HEL-LO-`
- D. `-HELLO`

**Answer:** C. `HEL-LO-` — $\mathcal B$ merges repeats, then deletes blanks. `HEL-LO-` merges nothing and drops blanks to give HELLO. `HHEELLLOO` merges to HELO; `HE-LLO` merges the LL to give HELO; `-HELLO` merges LL too. A doubled letter needs a blank between its copies.

**Question:** A target has $U=3$ labels with no adjacent repeats. With $T=8$ frames, how many alignments does CTC sum over?

- A. 8
- B. 56
- C. 462
- D. $4^8=65536$

**Answer:** C. 462 — An alignment is 3 label runs (length $\ge1$) interleaved with 4 blank runs (length $\ge0$) totalling 8 frames, which gives $\binom{T+U}{2U}=\binom{11}{6}=462$. $4^8$ counts every path over a 4-symbol alphabet, most of which collapse to other labellings.

**Key points**

- CTC adds a blank $\varepsilon$ and defines $p(y\mid x)$ as the sum of $\prod_t y_t(\pi_t)$ over every path $\pi$ with $\mathcal B(\pi)=y$.
- $\mathcal B$ merges repeats **then** removes blanks; a blank between equal labels is the only way to emit a doubled letter.
- It needs $T\ge U+$ (number of adjacent repeats), assumes monotonic alignment and conditionally independent frames.
- There are $\binom{T+U}{2U}$ alignments for a repeat-free target; the sum is done by dynamic programming (see **CTC loss**).
- Trained models are peaky: blank almost everywhere, labels as spikes.
- Greedy finds the best path, not the best labelling; prefix beam search sums paths per prefix and can add an LM via shallow fusion.
- Attention models add label dependencies but can loop or skip; RNN-T keeps CTC's monotonic alignment and adds a prediction network.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
