Definition
Connectionist Temporal Classification (CTC) is a way to train and decode a network that reads a long input sequence (audio frames, image columns, pen strokes) and must output a shorter label sequence , , without knowing which frames belong to which label. The network emits a distribution over the labels plus a special blank at every frame; a frame-level path is mapped to a label sequence by merging repeats and deleting blanks, and is the total probability of every path that maps to .
Speech is the classic case. One second of audio is 100 feature frames at a 10 ms hop, but the transcript "cat" is three characters, and nobody has marked where the "c" ends and the "a" begins. Before CTC, recognisers needed that segmentation: an HMM produced frame-level targets by forced alignment, and the neural network was trained as a frame classifier on them. CTC removes the alignment step: the network is trained directly on (input, transcript) pairs.
It was introduced by Alex Graves, Santiago Fernández, Faustino Gomez and Jürgen Schmidhuber at ICML 2006 ("Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks"), with bidirectional LSTMs on TIMIT phoneme recognition. Graves and colleagues then used it for unconstrained handwriting recognition (2009), Baidu's Deep Speech (Hannun et al., 2014) built an end-to-end speech recogniser on it, and the CRNN of Shi, Bai and Yao (2015) made it the standard loss for scene-text OCR. Self-supervised speech encoders such as wav2vec 2.0 (Baevski et al., 2020) are still usually fine-tuned with a CTC head.
This article covers the model: the blank, the collapse map, the alignment set, the assumptions and decoding. How and its gradient are actually computed (the forward–backward recursion, PyTorch's nn.CTCLoss) is the subject of CTC loss.
The blank and the collapse map B
Let be the label alphabet and . At each frame the network outputs a softmax over . A path (or alignment) is a sequence with one symbol per frame. The many-to-one map turns a path into a labelling in two steps:
B(π): collapse a path
- 1Merge consecutive repeats:
CC-AA--TbecomesC-A-T. A label held for several frames counts once. - 2Delete every blank:
C-A-TbecomesCAT. - 3The order matters.
CA-ATmerges nothing (the two A's are separated by a blank) and givesCAAT;CAATas a path merges toCAT. A blank between two equal labels is the only way to emit a doubled letter.
The blank does two jobs. It lets the network say "nothing new here" on the many frames between labels (silence, transitions, the inside of a long vowel), and it separates genuine repeats such as the "ll" in "hello". Without it, the network could never output "aa" at all.
The probability of a labelling
- network softmax output for symbol at frame
- a path: one symbol of per frame,
- the set of all paths that collapse to the target , its alignments
- conditional independence: given , the frame outputs are independent
Training maximises , i.e. minimises . The set of alignments is large: for a target of labels with no adjacent repeats there are exactly of them (a path is label runs of length interleaved with blank runs of length ). "CAT" has exactly 1 alignment in 3 frames (CAT itself) and 7 in 4 frames; in 8 frames it has ; at frames there are about . The sum is never enumerated: a dynamic programme over the blank-extended target computes it in (see CTC loss).
At a glance
- Task
- sequence labelling with and monotonic alignment: speech recognition, handwriting, OCR, sign language, keyword spotting
- Output layer
- per-frame softmax over symbols (labels plus blank); blank is index 0 by convention
- Supervision
- only the label sequence; no frame boundaries
- Length constraint
- , where is the number of adjacent equal labels in (each needs a blank between)
- Alignments
- for labels without repeats; fewer with repeats (e.g. 165 for "CAAT" vs 462 for "CAT" at )
- Assumption
- frame outputs independent given ; no model of label-to-label dependencies inside the loss
- Decoding
- greedy best path (fast, approximate), prefix beam search, beam search with an external language model
Interactive: alignments, collapse and decoding
Worked example: two frames
Take and a one-letter alphabet , with , at both frames. Every path and where sends it:
| Path π | B(π) | p(π|x) |
|---|---|---|
-- | ∅ (empty) | |
-a | a | |
a- | a | |
aa | a |
The single most probable path is -- (0.36), so greedy decoding outputs the empty string. But : the labelling "a" is almost twice as likely, because its mass is split over three alignments. "aa" is impossible here (), since it needs at least 3 frames (a-a). This gap between the most probable path and the most probable labelling is what beam search exists to close.
Assumptions and their consequences
Monotonic alignment. Paths move left to right through the target and can never go back. That is right for speech and handwriting, wrong for translation, where word order changes.
Conditional independence. : once the encoder has seen the whole input, each frame's choice ignores the choices at other frames. A bidirectional or Transformer encoder can still use plenty of context to produce , but the loss cannot express "after 'q' comes 'u'". Character CTC models therefore make phonetically plausible spelling errors, and an external language model helps a lot.
Length. Each label needs at least one frame, and each adjacent repeat needs an extra blank frame: . Encoders that downsample (convolutional front ends reduce frame rate by 4–8× in modern ASR) can violate this for fast speech or long transcripts, and the loss becomes infinite (see CTC loss for zero_infinity).
Decoding
The ideal output is , a search over labellings in which each candidate is itself a sum over alignments. It is intractable in general, so three approximations are used.
Where it's used
Greedy (best path)
Take at every frame, then apply . , no search. It finds the most probable path, which is not necessarily the most probable labelling (the two-frame example, and the 'ambiguous' preset in the figure). On peaky, well-trained models the dominant path usually carries most of the mass and the two agree.
Prefix beam search
Keep the best label prefixes, not paths. Each prefix stores two numbers, the probability of all paths ending in blank, , and ending in its last label, , so that at the next frame a repeated label can be merged (extends from ) or appended as a new label (only from ). Paths that collapse to the same prefix are summed instead of competing.
Beam search with an LM (shallow fusion)
Score prefixes by , where the LM is an n-gram or neural model over characters or words, is its weight and a length (word-insertion) bonus that offsets the LM's preference for short outputs. Deep Speech used exactly this form. WFST decoders (as in EESEN and Kaldi-style pipelines) compile lexicon and grammar into the search graph.
1import torch2from collections import defaultdict34def greedy_decode(log_probs, blank=0): # log_probs: (T, C) for one utterance5 best = log_probs.argmax(dim=-1).tolist() # best path, one symbol per frame6 out, prev = [], None7 for k in best: # B: merge repeats, then drop blanks8 if k != prev and k != blank:9 out.append(k)10 prev = k11 return out1213def prefix_beam_search(probs, width=8, blank=0): # probs: (T, C) softmax outputs14 beam = {(): (1.0, 0.0)} # prefix -> (p_blank_end, p_label_end)15 for row in probs.tolist():16 nxt = defaultdict(lambda: [0.0, 0.0])17 for prefix, (pb, pnb) in beam.items():18 nxt[prefix][0] += (pb + pnb) * row[blank] # emit blank19 for k, pk in enumerate(row):20 if k == blank:21 continue22 if prefix and prefix[-1] == k:23 nxt[prefix][1] += pnb * pk # repeat: merged24 nxt[prefix + (k,)][1] += pb * pk # after a blank: new label25 else:26 nxt[prefix + (k,)][1] += (pb + pnb) * pk27 beam = dict(sorted(nxt.items(), key=lambda kv: -sum(kv[1]))[:width])28 return max(beam.items(), key=lambda kv: sum(kv[1]))2930# the two-frame example: blank 0.6, "a" 0.4 at both frames31probs = torch.tensor([[0.6, 0.4], [0.6, 0.4]])32print(greedy_decode(probs.log())) # [] best path "--"33best, (pb, pnb) = prefix_beam_search(probs)34print(best, pb + pnb) # (1,) ≈0.64 -> "a": 0.24 ending in blank + 0.40 ending in "a"In production use a tested implementation: torchaudio.models.decoder.ctc_decoder (a lexicon- and KenLM-aware beam search from Flashlight) or pyctcdecode. Probabilities in real models should be handled in log space; the toy above multiplies raw probabilities for readability.
CTC versus attention and transducers
| Aspect | CTC | Attention encoder–decoder | RNN-Transducer |
|---|---|---|---|
| Alignment | monotonic, marginalised | learned soft attention, any order | monotonic, marginalised |
| Label dependencies | none (conditionally independent) | full, autoregressive decoder | yes, via a prediction network |
| Output length | required | unrestricted | unrestricted (emits labels per frame) |
| Inference | one parallel pass + collapse | sequential, beam search | sequential over frames and labels |
| Streaming | natural | hard (attends to the whole input) | natural |
| Typical failure | spelling errors without an LM | looping, skipping, hallucinated words on long inputs | memory-hungry loss over a T×U×V lattice |
The attention route is Listen, Attend and Spell (Chan et al., 2016); the transducer is Graves (2012), "Sequence Transduction with Recurrent Neural Networks", which removes CTC's independence assumption by conditioning each output on previous labels. In practice they are often combined: joint CTC–attention training (Kim, Hori and Watanabe, 2017) uses a CTC head as an auxiliary loss that forces the attention to stay monotonic, and decoding mixes both scores.
Use when
- +Trains on transcripts alone: no frame-level segmentation or forced alignment needed.
- +Non-autoregressive: one forward pass gives every frame's output, so greedy decoding is fast and trivially parallel.
- +Monotonic alignment is a strong, correct prior for speech, handwriting and OCR; it cannot loop or skip like attention can.
- +Streams naturally with a causal or chunked encoder.
- +Plays well with external language models and lexicons at decoding time.
Avoid when
- −Conditional independence: no output-side language model inside the loss, so accuracy depends on the encoder and an external LM.
- −Needs : unusable when the output can be longer than the input, or for non-monotonic tasks such as translation.
- −Peaky, blank-dominated outputs give unreliable timing and phone boundaries.
- −Most probable path ≠ most probable labelling; exact decoding is intractable.
Check yourself
Which path collapses to "HELLO" under ?
Check yourself
A target has labels with no adjacent repeats. With frames, how many alignments does CTC sum over?
Remember
- ✓CTC adds a blank and defines as the sum of over every path with .
- ✓ merges repeats then removes blanks; a blank between equal labels is the only way to emit a doubled letter.
- ✓It needs (number of adjacent repeats), assumes monotonic alignment and conditionally independent frames.
- ✓There are alignments for a repeat-free target; the sum is done by dynamic programming (see CTC loss).
- ✓Trained models are peaky: blank almost everywhere, labels as spikes.
- ✓Greedy finds the best path, not the best labelling; prefix beam search sums paths per prefix and can add an LM via shallow fusion.
- ✓Attention models add label dependencies but can loop or skip; RNN-T keeps CTC's monotonic alignment and adds a prediction network.