DeepSpeech

Baidu's end-to-end recogniser: an RNN trained with CTC maps spectrograms to characters, and a beam search maximises log p_ctc + α log p_lm + β·words. Deep Speech 2 added convolutions, GRUs and BatchNorm.

8 min read813 wordscore0 viewsupdated 2026-10-15

Definition

DeepSpeech names two Baidu Research systems and an open-source descendant. Deep Speech (Hannun, Case, Casper, Catanzaro, Diamos, Elsen, Prenger, Satheesh, Sengupta, Coates and Ng, 2014, "Deep Speech: Scaling up end-to-end speech recognition") is a 5-layer RNN trained with CTC from spectrograms straight to characters. Deep Speech 2 (Amodei et al., ICML 2016, "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin") is a deeper convolutional-recurrent version. Mozilla DeepSpeech is an open-source TensorFlow engine based on the first paper.

In 2014 the best recognisers were hybrid pipelines. A DNN classified frames into HMM states. Those state targets came from a forced alignment made by an earlier GMM system. A pronunciation lexicon mapped phones to words, and speaker adaptation and hand-designed features sat on top. Each part was trained separately.

Deep Speech replaced the acoustic model, the lexicon and the alignments with one network. It reads spectrogram frames and outputs a character distribution per frame, trained with Connectionist Temporal Classification, so no frame labels or phonemes are needed. Its other bet was scale: thousands of hours of speech, multi-GPU training, and synthetic noise to make the model robust.

Deep Speech (2014): five layers and CTC

The input at time tt is a spectrogram frame together with CC neighbouring frames on each side (C∈{5,7,9}C\in\{5,7,9\}). Three fully connected layers come first. They use a clipped ReLU, g(z)=min⁡(max⁡(z,0),20)g(z)=\min(\max(z,0),20), which keeps activations bounded in the recurrent layer. Layer 4 is a bidirectional recurrent layer. It is a plain RNN rather than an LSTM, chosen because it is cheaper to compute and to store. Layer 5 is fully connected, followed by a softmax over the characters a–z, space, apostrophe and the CTC blank.

ht(f)=g(W(4)ht(3)+Wr(f)ht−1(f)+b(4)),ht(b)=g(W(4)ht(3)+Wr(b)ht+1(b)+b(4)),ht(5)=g(W(5)(ht(f)+ht(b))+b(5))h^{(f)}_t=g\big(W^{(4)}h^{(3)}_t+W^{(f)}_r h^{(f)}_{t-1}+b^{(4)}\big),\quad h^{(b)}_t=g\big(W^{(4)}h^{(3)}_t+W^{(b)}_r h^{(b)}_{t+1}+b^{(4)}\big),\quad h^{(5)}_t=g\big(W^{(5)}(h^{(f)}_t+h^{(b)}_t)+b^{(5)}\big)
ht(3)h^{(3)}_t
output of the third fully connected layer at frame tt
h(f), h(b)h^{(f)},\,h^{(b)}
forward recurrence (left to right) and backward recurrence (right to left); they are summed, not concatenated
gg
clipped ReLU min⁡(max⁡(z,0),20)\min(\max(z,0),20)
Deep Speech (Hannun et al., 2014)Deep Speech 2 (Amodei et al., 2016)spectrogram frame t plus C frames each sideh1–h3: 3 × fully connectedclipped ReLU min(max(x, 0), 20)h4: bidirectional RNNforward → and backward ← states, summedh5: fully connectedsoftmax over charactersa–z, space, apostrophe, blankwhole spectrogram (time × frequency)1–3 × 2D convolutiontime × frequency, striding in timeup to 7 × bidirectional GRU / RNNsequence-wise BatchNorm on inputsfully connectedsoftmax: English chars or MandarinChinese characters, plus blanktrain: CTC loss, summed over every alignment of the transcriptdecode: beam search maximising Q(c) = log P(c | x) + α log P_lm(c) + β · words(c)DS1: ~5,000 h own read speech + Switchboard, Fisher, WSJ · noise added in synthesisDS2: 11,940 h English, 9,400 h Mandarin · SortaGrad: first epoch sorted by length
fig.Deep Speech (left): a spectrogram frame with C frames of context on each side, three clipped-ReLU fully connected layers, one bidirectional RNN whose two directions are summed, a fully connected layer and a character softmax. Deep Speech 2 (right): 2D convolutions over time and frequency, up to seven bidirectional GRU or RNN layers with BatchNorm, then a softmax over English characters or Chinese characters. Both are trained with CTC and decoded by beam search with an n-gram LM weighted by α and a word bonus β.

The training data were about 5,000 h of read speech from 9,600 speakers, collected by Baidu, plus Switchboard, Fisher and WSJ. To make the model robust to noise, clean utterances were mixed with many noise tracks. Some speakers were also recorded while hearing loud noise in headphones, which captures the Lombard effect: people raise and change their voice in noise. On the full Switchboard Hub5'00 test set, a model trained on the 2,300 h of Switchboard and Fisher reached 16.0% WER.

Deep Speech 2 (2016): deeper, faster, two languages

Deep Speech 2 kept CTC and the character outputs but rebuilt the network. One to three 2D convolutions over time and frequency come first, with striding in time to shorten the sequence. Then come up to seven bidirectional recurrent layers (see Gated recurrent unit (GRU)). Batch normalization is applied only to the input-to-hidden connections and normalised over the whole sequence. The same architecture works for English and Mandarin; for Mandarin it outputs Chinese characters directly.

Two training tricks made the deep model trainable. SortaGrad is a simple curriculum: in the first epoch, mini-batches are taken in order of increasing utterance length, and later epochs are random. Long utterances have larger CTC losses and gradients early in training, so starting short is more stable. For streaming, a unidirectional model adds a small lookahead convolution over a few future frames instead of the backward RNN. Training used 11,940 h of English and 9,400 h of Mandarin.

Reference numbers

Deep Speech
5 layers (3 FC, 1 bidirectional RNN, 1 FC); clipped ReLU at 20; characters + blank; about 5,000 h of in-house read speech; 16.0% WER on Hub5'00 (full)
Deep Speech 2
up to 3 conv + 7 recurrent layers; sequence-wise BatchNorm; SortaGrad; 11,940 h English and 9,400 h Mandarin
DS2 on LibriSpeech
5.33% WER test-clean, 13.25% test-other, close to the human transcribers measured in the same paper (5.83% and 12.69%)
Decoder
CTC prefix beam search with an n-gram LM: Q=log⁡Pctc+αlog⁡Plm+β wordsQ=\log P_{\text{ctc}}+\alpha\log P_{\text{lm}}+\beta\,\text{words}; α,β\alpha,\beta tuned on a dev set

Decoding with a language model

A character CTC model spells phonetically: on its own it writes "bostin" or "cabtoboston" when the sound is ambiguous. Both papers therefore decode with a beam search over character prefixes (prefix beam search; see Connectionist Temporal Classification and Beam search). An n-gram language model is consulted each time a word is completed.

Q(c)=log⁡Pctc(c∣x)+α log⁡Plm(c)+β word_count(c)Q(c)=\log P_{\text{ctc}}(c\mid x)+\alpha\,\log P_{\text{lm}}(c)+\beta\,\mathrm{word\_count}(c)
Pctc(c∣x)P_{\text{ctc}}(c\mid x)
sum over all frame paths that collapse to the character string cc, kept per prefix as blank-ending and label-ending parts
α\alpha
language-model weight: how much to trust the text prior over the acoustics
β\beta
word insertion bonus. Every word multiplies in an LM probability below 1, so without β\beta the search prefers fewer words
acoustic model output p(k | x, t): 21 frames × 10 symbols (_ = blank, ␣ = space)_␣abcinost_aa␣c_ab␣_␣to␣bostin_greedy (argmax per frame, merge, drop _): “a cab to bostin”word bigram LM⟨s⟩ a 0.3a cab 0.1⟨s⟩ cab 0.1cab to 0.4to boston 0.3else 0.4·p(w)OOV p(w) 1e-4p(a) = p(to) .25p(cab) = .05Q = log p_ctc + α·log p_lm + β·|words| · α = 0.8, β = 1.00, beam 8hypothesislog p_ctcα·log p_lmβ·|W|Q1. “a cab to boston”-4.29-4.50+4.00-4.792. “cab to boston”-5.45-3.54+3.00-5.993. “a cab a to boston”-6.01-7.45+5.00-8.474. “a cab to bosto”-7.51-11.64+4.00-15.155. “cab to bosto”-8.68-10.68+3.00-16.35decoded = reference: the LM fixed “bostin” and kept the weak “a”
fig.A 21-frame CTC output for "a cab to boston" with three planted ambiguities: a weak "a" (0.50 vs blank 0.42), a noise blip after "cab" (a = 0.2), and "i" (0.52) beating "o" (0.38) in "boston". Greedy decoding gives "a cab to bostin". At the default α = 0.8, β = 1, beam 8, the winner is the reference, with Q = −4.29 − 4.50 + 4.00 = −4.79; "cab to boston" is second at −5.99. With α = 0 the acoustics win and the result is "bostin" (Q = −4.00 at β = 0). With α = 1.5 and β = 0 the LM deletes the weak "a", giving "cab to boston". At α = 0.8 and β = 5 the bonus inserts a word: "a cab a to boston". At α = 0 and β = 4 the result is the nonsense "a c ab a to b st n".
Hypothesislog P_ctclog P_lmα·log P_lmβ·wordsQ
a cab to boston−4.29−1.20 − 2.30 − 0.92 − 1.20 = −5.63−4.50+4.00−4.79
cab to boston−5.45−2.30 − 0.92 − 1.20 = −4.42−3.54+3.00−5.99
a cab to bostin−4.00−1.20 − 2.30 − 0.92 − 10.13 = −14.55−11.64+4.00−11.64
Worked example at α = 0.8, β = 1 (natural logs; bigram LM: p(a|⟨s⟩) = 0.3, p(cab|a) = 0.1, p(cab|⟨s⟩) = 0.1, p(to|cab) = 0.4, p(boston|to) = 0.3, out-of-vocabulary words backed off to 0.4 × 10⁻⁴)

"bostin" has the best acoustic score but is out of vocabulary, and its ln⁡(0.4×10−4)=−10.13\ln(0.4\times10^{-4})=-10.13 sinks it. Dropping "a" saves 0.8×1.20=0.960.8\times1.20=0.96 of LM cost. However, it loses 1.161.16 of acoustic score and the β=1\beta=1 word bonus, so the reference wins by 1.201.20.

python
1import soundfile as sf
2from deepspeech import Model
3
4ds = Model("deepspeech-0.9.3-models.pbmm") # acoustic model (TensorFlow graph)
5ds.enableExternalScorer("deepspeech-0.9.3-models.scorer") # KenLM n-gram + vocabulary trie
6ds.setScorerAlphaBeta(0.93, 1.18) # alpha = LM weight, beta = word bonus
7ds.setBeamWidth(500)
8
9audio, sr = sf.read("cab.wav", dtype="int16") # 16 kHz mono PCM
10assert sr == ds.sampleRate()
11print(ds.stt(audio))
12
13ds.disableExternalScorer() # acoustic-only decoding, for comparison
14print(ds.stt(audio))
Mozilla DeepSpeech inference: acoustic model plus KenLM scorer, with α and β set explicitly (not run here; the deepspeech package is archived and needs an old Python)

Use when

  • +One network from spectrogram to characters: no lexicon, phone set or forced alignment
  • +CTC is non-autoregressive; DS2's lookahead variant streams
  • +An external LM can be swapped in for a new domain without retraining the acoustic model

Avoid when

  • −CTC frames are conditionally independent, so good spelling depends on the external LM
  • −α and β need tuning and interact with beam width and vocabulary
  • −Needs thousands of labelled hours; later self-supervised and attention models need far less labelled data

Check yourself

You raise α and the decoder starts dropping short words such as "a" and "the". Which change targets this directly?

Check yourself

What is SortaGrad?

Remember

  • ✓Deep Speech (2014): spectrogram → 3 clipped-ReLU FC layers → bidirectional RNN → FC → characters, trained with CTC.
  • ✓Scale and synthesis: about 5,000 h of in-house speech, noise superposition and Lombard-effect recordings; 16.0% WER on Hub5'00.
  • ✓Deep Speech 2 (2016): conv front end, up to 7 bidirectional GRU/RNN layers, sequence-wise BatchNorm, SortaGrad; English and Mandarin.
  • ✓Decode: Q=log⁡Pctc+αlog⁡Plm+β⋅wordsQ=\log P_{\text{ctc}}+\alpha\log P_{\text{lm}}+\beta\cdot\text{words}; α\alpha fixes spelling, β\beta offsets the LM's bias toward fewer words.
  • ✓Mozilla DeepSpeech made the design open source: a streaming LSTM variant with a KenLM scorer.
← back to Deep Learning roadmap