Definition
Optical character recognition (OCR) is the task of turning an image of text into a character string. Deep OCR is usually two stages: a text detector finds word or line regions, and a text recogniser reads each cropped region. The reference recogniser is CRNN (Shi, Bai and Yao, "An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition", TPAMI 2017). It is a CNN, then a bidirectional LSTM, trained with Connectionist Temporal Classification (Graves et al., ICML 2006).
Input: a scanned page, a photo of a street sign, a receipt. Output: the text, usually with a box or polygon for each word or line. Scanned documents are the easy case: dark text on a clean background, aligned and of uniform size. Scene text is hard: arbitrary fonts, perspective, curvature, blur, low contrast and clutter.
Classical engines binarised the page, segmented it into lines and single characters, and classified each glyph. Character segmentation is the weak step: touching or broken characters and cursive handwriting cannot be cut reliably. The deep approach never segments characters. It reads a whole word or line as a sequence, which is also why Tesseract 4 (2018) switched to an LSTM recogniser.
Stage 1: text detection
Text boxes are long, thin, rotated and sometimes curved, so generic box detectors such as Faster R-CNN fit them poorly. Two detectors became standard. EAST (Zhou et al., CVPR 2017) is a fully convolutional network that predicts, for every pixel, a text score and a geometry: either a rotated box (4 distances to the box edges plus an angle) or a quadrilateral (8 corner offsets). Locality-aware NMS then merges the per-pixel boxes.
DBNet (Liao et al., "Real-time Scene Text Detection with Differentiable Binarization", AAAI 2020) is a segmentation detector. A ResNet with an FPN predicts a text probability map and a per-pixel threshold map at 1/4 resolution. Thresholding with a hard step has no gradient, so DBNet trains an approximate binary map instead:
- predicted probability that pixel (i, j) lies inside a (shrunk) text region
- learned threshold, supervised to be high near text borders, which separates adjacent words
- amplification factor; with the sigmoid is close to a step but still differentiable
At test time the threshold branch can be dropped: binarise , take connected components, and dilate each shrunk region back to full size. Because it is pure segmentation, DBNet handles curved text, which a 4-corner box cannot.
Stage 2: recognition as sequence labelling
CRNN resizes a word crop to height 32 (100 × 32 in the paper). Its 7 convolutional layers and max-pools shrink the height to 1 while keeping horizontal resolution, so the feature map becomes a left-to-right sequence of column vectors, each seeing a vertical slice of the word. A 2-layer Bidirectional LSTM with 256 units per direction reads the sequence, and a softmax per frame gives a distribution over the alphabet plus a blank .
The labels are only the string, for example "hello", not which frame shows which letter. CTC removes the need for that alignment. Define a collapse map that merges repeated symbols and then deletes blanks, so . Then sum over every frame path that collapses to the label:
- a frame-level path of length over the alphabet plus
- all paths that collapse to ; for "hello" in 12 frames there are 8008
- softmax output at frame ; frames are conditionally independent given
The sum is computed in by the forward algorithm over the blank-extended label . A double letter needs a blank between its copies, otherwise merges them, so CTC needs . Decoding is either greedy (argmax per frame, then ) or beam search, optionally with a lexicon or language model.
Reference systems
- CRNN (TPAMI 2017)
- grey input 100 × 32; 7 conv layers + 2 BiLSTM layers (256 units); 37 classes (26 letters + 10 digits + blank); 8.3M parameters. Lexicon-free word accuracy: IIIT5K 78.2%, SVT 80.8%, IC03 89.4%, IC13 86.7%
- EAST (CVPR 2017)
- per-pixel score map + rotated-box or quadrilateral geometry; locality-aware NMS; one network, no proposals
- DBNet (AAAI 2020)
- ResNet-18 or ResNet-50 + FPN; maps at 1/4 resolution; differentiable binarisation with
- TrOCR (Li et al., 2021)
- ViT-style image encoder (BEiT or DeiT init) + text Transformer decoder (RoBERTa or MiniLM init); 384 × 384 input, 16 × 16 patches; autoregressive subword output, no CTC
- PP-OCR (Du et al., 2020)
- production pipeline: DB detector → text-direction classifier → CRNN recogniser
Worked example: CTC on three frames
Alphabet , target "a", , with and . Of the paths, six collapse to "a" (aεa collapses to "aa", εεε to the empty string).
| Path π | ∏ p_t(π_t) | Probability |
|---|---|---|
| a a ε | 0.6 × 0.7 × 0.8 | 0.336 |
| ε a ε | 0.4 × 0.7 × 0.8 | 0.224 |
| a ε ε | 0.6 × 0.3 × 0.8 | 0.144 |
| a a a | 0.6 × 0.7 × 0.2 | 0.084 |
| ε a a | 0.4 × 0.7 × 0.2 | 0.056 |
| ε ε a | 0.4 × 0.3 × 0.2 | 0.024 |
| sum = p(a | x) | 0.868, loss −ln 0.868 = 0.142 |
The best path a a ε holds only 0.336. Training on it alone (a hard alignment) would ignore 61% of the probability mass on the right answer; CTC pushes up all six. The remaining mass goes to "" (0.096) and "aa" (0.036).
Transformers and OCR-free models
TrOCR (Li et al., Microsoft, 2021; AAAI 2023) drops the CNN and CTC. A ViT-style encoder reads 16 × 16 patches of the text-line image, and an autoregressive Transformer decoder generates subword tokens through Cross-attention, like an image captioner. Both halves start from pre-trained models and are pre-trained further on large synthetic printed and handwritten text. Autoregressive decoding can model the language of the text, which CTC's frame independence cannot, but it is slower and can produce fluent, hallucinated text on unreadable crops.
Donut (Kim et al., "OCR-free Document Understanding Transformer", ECCV 2022) goes further: a Swin encoder and a BART decoder map a whole document image directly to structured output (for example JSON fields of a receipt), with no separate detector or recogniser.
Measuring OCR
Recognition is scored by character error rate, : substitutions, deletions and insertions of the minimum edit (Levenshtein) path, divided by the number of reference characters. WER is the same at the word level. Both can exceed 1, since insertions are unbounded. Example: reference "open 24 hours", output "opem 24 hour". That is 2 character edits over 13 characters, CER 0.154, but 2 of 3 words wrong, WER 0.667. Detection uses precision, recall and their harmonic mean (H-mean), with a match at IoU ≥ 0.5 in the ICDAR 2015 protocol.
1import torch, torch.nn as nn23alphabet = "-0123456789abcdefghijklmnopqrstuvwxyz" # index 0 = blank4T, N, C = 26, 2, len(alphabet) # 26 frames, batch of 2, 37 classes5logits = model(images) # (T, N, C) from CNN + BiLSTM + linear6log_probs = logits.log_softmax(2)78targets = torch.tensor([alphabet.index(c) for c in "hello" + "open"]) # concatenated labels9target_lengths = torch.tensor([5, 4])10input_lengths = torch.full((N,), T, dtype=torch.long)11loss = nn.CTCLoss(blank=0, zero_infinity=True)(log_probs, targets, input_lengths, target_lengths)12loss.backward()1314def greedy_decode(lp): # lp: (T, C) for one sample15 best = lp.argmax(1).tolist()16 out = [k for t, k in enumerate(best) if k != 0 and (t == 0 or k != best[t - 1])]17 return "".join(alphabet[k] for k in out)1819def cer(hyp, ref): # Levenshtein distance / len(ref)20 d = list(range(len(ref) + 1))21 for i, h in enumerate(hyp, 1):22 prev, d[0] = d[:], i23 for j, c in enumerate(ref, 1):24 d[j] = min(prev[j] + 1, d[j - 1] + 1, prev[j - 1] + (h != c))25 return d[-1] / len(ref)2627print(cer("opem 24 hour", "open 24 hours")) # 0.1538...Use when
- +CTC recognisers are fast, parallel over frames and need only string labels
- +Attention or Transformer decoders (TrOCR) model the language of the text and handle irregular layouts
- +OCR-free models (Donut) remove error accumulation between detector, recogniser and parser
Avoid when
- −CTC assumes frames are independent given the image, so it has no built-in language model
- −Autoregressive decoding is sequential and can hallucinate plausible text on unreadable crops
- −Two-stage pipelines fail when detection misses or splits a word; the recogniser never sees it
Check yourself
A CTC recogniser's argmax per frame is ε h h e ε l l l o ε. What does greedy decoding output?
Check yourself
Why does CTC sum over all alignments instead of training on one fixed alignment?
Remember
- ✓OCR = text detection (EAST: per-pixel geometry; DBNet: probability map + learned threshold, B = σ(50(P − T))) + text recognition.
- ✓CRNN: CNN collapses height to 1 → sequence of frames → 2-layer BiLSTM → per-frame softmax with a blank ε.
- ✓CTC: , computed by the forward algorithm; decode by merging repeats and dropping blanks.
- ✓Repeated letters need a blank between them, and the label must fit in the available frames.
- ✓TrOCR replaces CNN + CTC with a ViT encoder and autoregressive decoder; Donut maps a document image straight to structured text.
- ✓Score recognition with CER and WER (edit distance over reference length), detection with IoU-matched H-mean.