Optical character recognition (OCR)

Deep OCR detects text regions (EAST, DBNet), then reads each crop as a sequence: CRNN (Shi et al., 2017) runs a CNN and BiLSTM and is trained with CTC, which sums over all frame alignments; TrOCR swaps in a transformer. Scored by CER and WER.

8 min read1005 wordscore0 viewsupdated 2026-10-15

Definition

Optical character recognition (OCR) is the task of turning an image of text into a character string. Deep OCR is usually two stages: a text detector finds word or line regions, and a text recogniser reads each cropped region. The reference recogniser is CRNN (Shi, Bai and Yao, "An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition", TPAMI 2017). It is a CNN, then a bidirectional LSTM, trained with Connectionist Temporal Classification (Graves et al., ICML 2006).

Input: a scanned page, a photo of a street sign, a receipt. Output: the text, usually with a box or polygon for each word or line. Scanned documents are the easy case: dark text on a clean background, aligned and of uniform size. Scene text is hard: arbitrary fonts, perspective, curvature, blur, low contrast and clutter.

Classical engines binarised the page, segmented it into lines and single characters, and classified each glyph. Character segmentation is the weak step: touching or broken characters and cursive handwriting cannot be cut reliably. The deep approach never segments characters. It reads a whole word or line as a sequence, which is also why Tesseract 4 (2018) switched to an LSTM recogniser.

Stage 1: text detection

Text boxes are long, thin, rotated and sometimes curved, so generic box detectors such as Faster R-CNN fit them poorly. Two detectors became standard. EAST (Zhou et al., CVPR 2017) is a fully convolutional network that predicts, for every pixel, a text score and a geometry: either a rotated box (4 distances to the box edges plus an angle) or a quadrilateral (8 corner offsets). Locality-aware NMS then merges the per-pixel boxes.

DBNet (Liao et al., "Real-time Scene Text Detection with Differentiable Binarization", AAAI 2020) is a segmentation detector. A ResNet with an FPN predicts a text probability map PP and a per-pixel threshold map TT at 1/4 resolution. Thresholding PP with a hard step has no gradient, so DBNet trains an approximate binary map instead:

B^i,j=11+e−k (Pi,j−Ti,j),k=50\hat B_{i,j}=\frac{1}{1+e^{-k\,(P_{i,j}-T_{i,j})}},\qquad k=50
Pi,jP_{i,j}
predicted probability that pixel (i, j) lies inside a (shrunk) text region
Ti,jT_{i,j}
learned threshold, supervised to be high near text borders, which separates adjacent words
kk
amplification factor; with k=50k=50 the sigmoid is close to a step but still differentiable

At test time the threshold branch can be dropped: binarise PP, take connected components, and dilate each shrunk region back to full size. Because it is pure segmentation, DBNet handles curved text, which a 4-corner box cannot.

1 · detect: where is the text? (DBNet)HELLOOPEN 24himage H × W × 3ResNet + FPNfused featuresat H/4 × W/4prob. map Pthreshold map TB = 1/(1+e^(−k(P−T)))k = 50, differentiabletrain: loss on P, T, Btest: binarise P, takecontours, dilate → boxes2 · recognise each crop: what does it say? (CRNN + CTC)HELLOcrop, resize to 1×32×100CNN (VGG-style)7 conv layers; latepools shrink height512 × 1 × 26 → 26 framesBiLSTM ×2256 unitsper directionsoftmax26 × 3736 chars + εargmax per frameεεhhεeeεεllεεllεoooεεεεεεεmerge repeatsε h ε e ε l ε l ε o εdrop ε"hello"the ε between the two l's keeps them aparttraining: CTC loss −log p(y|x) sums over every frame alignment, so no per-character boxes are needed
fig.The two-stage pipeline. DBNet predicts a probability map P and a threshold map T, binarises with B = σ(k(P − T)) and returns word boxes. Each crop is resized to height 32 and passed through CRNN: 7 conv layers collapse the height to 1 and leave one 512-d column per horizontal frame (26 frames for a 100-pixel-wide crop), a 2-layer BiLSTM adds context, a 37-way softmax per frame (36 characters + blank ε) is trained with CTC and decoded by merging repeats and dropping blanks.

Stage 2: recognition as sequence labelling

CRNN resizes a word crop to height 32 (100 × 32 in the paper). Its 7 convolutional layers and max-pools shrink the height to 1 while keeping horizontal resolution, so the feature map becomes a left-to-right sequence of TT column vectors, each seeing a vertical slice of the word. A 2-layer Bidirectional LSTM with 256 units per direction reads the sequence, and a softmax per frame gives a distribution over the alphabet plus a blank ε\varepsilon.

The labels are only the string, for example "hello", not which frame shows which letter. CTC removes the need for that alignment. Define a collapse map B\mathcal B that merges repeated symbols and then deletes blanks, so B(ε h h e ε l l ε l o)=hello\mathcal B(\varepsilon\,h\,h\,e\,\varepsilon\,l\,l\,\varepsilon\,l\,o)=\text{hello}. Then sum over every frame path that collapses to the label:

p(y∣x)=∑π∈B−1(y) ∏t=1Tpt(πt∣x),LCTC=−log⁡p(y∣x)p(\mathbf y\mid\mathbf x)=\sum_{\pi\in\mathcal B^{-1}(\mathbf y)}\ \prod_{t=1}^{T}p_t(\pi_t\mid\mathbf x),\qquad \mathcal L_{\text{CTC}}=-\log p(\mathbf y\mid\mathbf x)
π\pi
a frame-level path of length TT over the alphabet plus ε\varepsilon
B−1(y)\mathcal B^{-1}(\mathbf y)
all paths that collapse to y\mathbf y; for "hello" in 12 frames there are 8008
pt(⋅∣x)p_t(\cdot\mid\mathbf x)
softmax output at frame tt; frames are conditionally independent given x\mathbf x

The sum is computed in O(T⋅∣y∣)O(T\cdot|\mathbf y|) by the forward algorithm over the blank-extended label ε h ε e ε l ε l ε o ε\varepsilon\,h\,\varepsilon\,e\,\varepsilon\,l\,\varepsilon\,l\,\varepsilon\,o\,\varepsilon. A double letter needs a blank between its copies, otherwise B\mathcal B merges them, so CTC needs T≥∣y∣+(number of repeated adjacent pairs)T\ge|\mathbf y|+(\text{number of repeated adjacent pairs}). Decoding is either greedy (argmax per frame, then B\mathcal B) or beam search, optionally with a lexicon or language model.

Reference systems

CRNN (TPAMI 2017)
grey input 100 × 32; 7 conv layers + 2 BiLSTM layers (256 units); 37 classes (26 letters + 10 digits + blank); 8.3M parameters. Lexicon-free word accuracy: IIIT5K 78.2%, SVT 80.8%, IC03 89.4%, IC13 86.7%
EAST (CVPR 2017)
per-pixel score map + rotated-box or quadrilateral geometry; locality-aware NMS; one network, no proposals
DBNet (AAAI 2020)
ResNet-18 or ResNet-50 + FPN; maps at 1/4 resolution; differentiable binarisation with k=50k=50
TrOCR (Li et al., 2021)
ViT-style image encoder (BEiT or DeiT init) + text Transformer decoder (RoBERTa or MiniLM init); 384 × 384 input, 16 × 16 patches; autoregressive subword output, no CTC
PP-OCR (Du et al., 2020)
production pipeline: DB detector → text-direction classifier → CRNN recogniser
CTC over 12 frames · {ε,h,e,l,o} · target "hello" · click a cell to move a peakimagehellot1t2t3t4t5t6t7t8t9t10t11t12ε0.820.080.040.030.860.050.040.770.040.060.020.81h0.060.770.800.060.040.030.050.070.030.030.040.06e0.030.050.070.800.020.050.080.040.020.070.040.03l0.050.070.040.040.050.850.760.050.890.050.020.05o0.040.030.050.070.030.030.070.070.020.790.880.05softmaxargmaxεhheεllεlooεmergeε h e ε l ε l o ε → drop ε → "hello"CERedit distance("hello", "hello") / 5 = 0/5 = 0.00p("hello"|x), all paths0.1730p(best path)0.0862CTC loss −log p("hello"|x) = 1.755 · sums 8008 alignments (5 labels, 12 frames)the best path carries 49.8% of p("hello"); the other alignments supply the rest
frame peaks
fig.A 12-frame output over {ε, h, e, l, o}. At the default (β = 3) greedy decoding gives "hello" with CER 0.00; p("hello" | x) = 0.173 (CTC loss 1.755), of which the single best path carries 0.0862, 49.8%. Press "no blank between the l's": greedy now reads "helo", CER 1/5 = 0.20, and p("hello") falls to 0.0184. Set β = 6 on the clean peaks: p("hello") = 0.906, loss 0.099. At β = 0 the frames are nearly uniform, the argmax reads "heolehleoh" and CER is 1.00. Click any cell to move that frame's peak.

Worked example: CTC on three frames

Alphabet {ε,a}\{\varepsilon, a\}, target "a", T=3T=3, with pt(a)=0.6, 0.7, 0.2p_t(a)=0.6,\,0.7,\,0.2 and pt(ε)=0.4, 0.3, 0.8p_t(\varepsilon)=0.4,\,0.3,\,0.8. Of the 23=82^3=8 paths, six collapse to "a" (aεa collapses to "aa", εεε to the empty string).

Path π∏ p_t(π_t)Probability
a a ε0.6 × 0.7 × 0.80.336
ε a ε0.4 × 0.7 × 0.80.224
a ε ε0.6 × 0.3 × 0.80.144
a a a0.6 × 0.7 × 0.20.084
ε a a0.4 × 0.7 × 0.20.056
ε ε a0.4 × 0.3 × 0.20.024
sum = p(a | x)0.868, loss −ln 0.868 = 0.142
The six alignments of "a" to 3 frames; their sum is p(a | x)

The best path a a ε holds only 0.336. Training on it alone (a hard alignment) would ignore 61% of the probability mass on the right answer; CTC pushes up all six. The remaining mass goes to "" (0.096) and "aa" (0.036).

Transformers and OCR-free models

TrOCR (Li et al., Microsoft, 2021; AAAI 2023) drops the CNN and CTC. A ViT-style encoder reads 16 × 16 patches of the text-line image, and an autoregressive Transformer decoder generates subword tokens through Cross-attention, like an image captioner. Both halves start from pre-trained models and are pre-trained further on large synthetic printed and handwritten text. Autoregressive decoding can model the language of the text, which CTC's frame independence cannot, but it is slower and can produce fluent, hallucinated text on unreadable crops.

Donut (Kim et al., "OCR-free Document Understanding Transformer", ECCV 2022) goes further: a Swin encoder and a BART decoder map a whole document image directly to structured output (for example JSON fields of a receipt), with no separate detector or recogniser.

Measuring OCR

Recognition is scored by character error rate, CER=(S+D+I)/N\text{CER}=(S+D+I)/N: substitutions, deletions and insertions of the minimum edit (Levenshtein) path, divided by the number of reference characters. WER is the same at the word level. Both can exceed 1, since insertions are unbounded. Example: reference "open 24 hours", output "opem 24 hour". That is 2 character edits over 13 characters, CER 0.154, but 2 of 3 words wrong, WER 0.667. Detection uses precision, recall and their harmonic mean (H-mean), with a match at IoU ≥ 0.5 in the ICDAR 2015 protocol.

python
1import torch, torch.nn as nn
2
3alphabet = "-0123456789abcdefghijklmnopqrstuvwxyz" # index 0 = blank
4T, N, C = 26, 2, len(alphabet) # 26 frames, batch of 2, 37 classes
5logits = model(images) # (T, N, C) from CNN + BiLSTM + linear
6log_probs = logits.log_softmax(2)
7
8targets = torch.tensor([alphabet.index(c) for c in "hello" + "open"]) # concatenated labels
9target_lengths = torch.tensor([5, 4])
10input_lengths = torch.full((N,), T, dtype=torch.long)
11loss = nn.CTCLoss(blank=0, zero_infinity=True)(log_probs, targets, input_lengths, target_lengths)
12loss.backward()
13
14def greedy_decode(lp): # lp: (T, C) for one sample
15 best = lp.argmax(1).tolist()
16 out = [k for t, k in enumerate(best) if k != 0 and (t == 0 or k != best[t - 1])]
17 return "".join(alphabet[k] for k in out)
18
19def cer(hyp, ref): # Levenshtein distance / len(ref)
20 d = list(range(len(ref) + 1))
21 for i, h in enumerate(hyp, 1):
22 prev, d[0] = d[:], i
23 for j, c in enumerate(ref, 1):
24 d[j] = min(prev[j] + 1, d[j - 1] + 1, prev[j - 1] + (h != c))
25 return d[-1] / len(ref)
26
27print(cer("opem 24 hour", "open 24 hours")) # 0.1538...
One CRNN-style training step with CTC loss, greedy decoding and CER (PyTorch; not run here, shapes follow the torch.nn.CTCLoss documentation)

Use when

  • +CTC recognisers are fast, parallel over frames and need only string labels
  • +Attention or Transformer decoders (TrOCR) model the language of the text and handle irregular layouts
  • +OCR-free models (Donut) remove error accumulation between detector, recogniser and parser

Avoid when

  • −CTC assumes frames are independent given the image, so it has no built-in language model
  • −Autoregressive decoding is sequential and can hallucinate plausible text on unreadable crops
  • −Two-stage pipelines fail when detection misses or splits a word; the recogniser never sees it

Check yourself

A CTC recogniser's argmax per frame is ε h h e ε l l l o ε. What does greedy decoding output?

Check yourself

Why does CTC sum over all alignments instead of training on one fixed alignment?

Remember

  • ✓OCR = text detection (EAST: per-pixel geometry; DBNet: probability map + learned threshold, B = σ(50(P − T))) + text recognition.
  • ✓CRNN: CNN collapses height to 1 → sequence of frames → 2-layer BiLSTM → per-frame softmax with a blank ε.
  • ✓CTC: p(y∣x)=∑π∈B−1(y)∏tpt(πt)p(\mathbf y\mid\mathbf x)=\sum_{\pi\in\mathcal B^{-1}(\mathbf y)}\prod_t p_t(\pi_t), computed by the forward algorithm; decode by merging repeats and dropping blanks.
  • ✓Repeated letters need a blank between them, and the label must fit in the available frames.
  • ✓TrOCR replaces CNN + CTC with a ViT encoder and autoregressive decoder; Donut maps a document image straight to structured text.
  • ✓Score recognition with CER and WER (edit distance over reference length), detection with IoU-matched H-mean.
← back to Deep Learning roadmap