# DeepSpeech

> Baidu's end-to-end recogniser: an RNN trained with CTC maps spectrograms to characters, and a beam search maximises log p_ctc + α log p_lm + β·words. Deep Speech 2 added convolutions, GRUs and BatchNorm.

Source: https://www.mlfoundry.in/learn/deepspeech  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**DeepSpeech:** names two Baidu Research systems and an open-source descendant. **Deep Speech** (Hannun, Case, Casper, Catanzaro, Diamos, Elsen, Prenger, Satheesh, Sengupta, Coates and Ng, 2014, "Deep Speech: Scaling up end-to-end speech recognition") is a 5-layer RNN trained with CTC from spectrograms straight to characters. **Deep Speech 2** (Amodei et al., ICML 2016, "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin") is a deeper convolutional-recurrent version. **Mozilla DeepSpeech** is an open-source TensorFlow engine based on the first paper.

In 2014 the best recognisers were hybrid pipelines. A DNN classified frames into HMM states. Those state targets came from a forced alignment made by an earlier GMM system. A pronunciation lexicon mapped phones to words, and speaker adaptation and hand-designed features sat on top. Each part was trained separately.

Deep Speech replaced the acoustic model, the lexicon and the alignments with one network. It reads spectrogram frames and outputs a character distribution per frame, trained with **Connectionist Temporal Classification**, so no frame labels or phonemes are needed. Its other bet was scale: thousands of hours of speech, multi-GPU training, and synthetic noise to make the model robust.

## Deep Speech (2014): five layers and CTC

The input at time $t$ is a spectrogram frame together with $C$ neighbouring frames on each side ($C\in\{5,7,9\}$). Three fully connected layers come first. They use a **clipped ReLU**, $g(z)=\min(\max(z,0),20)$, which keeps activations bounded in the recurrent layer. Layer 4 is a bidirectional recurrent layer. It is a plain RNN rather than an LSTM, chosen because it is cheaper to compute and to store. Layer 5 is fully connected, followed by a softmax over the characters a–z, space, apostrophe and the CTC blank.

$$
h^{(f)}_t=g\big(W^{(4)}h^{(3)}_t+W^{(f)}_r h^{(f)}_{t-1}+b^{(4)}\big),\quad h^{(b)}_t=g\big(W^{(4)}h^{(3)}_t+W^{(b)}_r h^{(b)}_{t+1}+b^{(4)}\big),\quad h^{(5)}_t=g\big(W^{(5)}(h^{(f)}_t+h^{(b)}_t)+b^{(5)}\big)
$$

- $h^{(3)}_t$: output of the third fully connected layer at frame $t$
- $h^{(f)},\,h^{(b)}$: forward recurrence (left to right) and backward recurrence (right to left); they are summed, not concatenated
- $g$: clipped ReLU $\min(\max(z,0),20)$

> Figure: Deep Speech (left): a spectrogram frame with C frames of context on each side, three clipped-ReLU fully connected layers, one bidirectional RNN whose two directions are summed, a fully connected layer and a character softmax. Deep Speech 2 (right): 2D convolutions over time and frequency, up to seven bidirectional GRU or RNN layers with BatchNorm, then a softmax over English characters or Chinese characters. Both are trained with CTC and decoded by beam search with an n-gram LM weighted by α and a word bonus β.

The training data were about 5,000 h of read speech from 9,600 speakers, collected by Baidu, plus Switchboard, Fisher and WSJ. To make the model robust to noise, clean utterances were mixed with many noise tracks. Some speakers were also recorded while hearing loud noise in headphones, which captures the Lombard effect: people raise and change their voice in noise. On the full Switchboard Hub5'00 test set, a model trained on the 2,300 h of Switchboard and Fisher reached 16.0% WER.

## Deep Speech 2 (2016): deeper, faster, two languages

Deep Speech 2 kept CTC and the character outputs but rebuilt the network. One to three 2D convolutions over time and frequency come first, with striding in time to shorten the sequence. Then come up to seven bidirectional recurrent layers (see **Gated recurrent unit (GRU)**). **Batch normalization** is applied only to the input-to-hidden connections and normalised over the whole sequence. The same architecture works for English and Mandarin; for Mandarin it outputs Chinese characters directly.

Two training tricks made the deep model trainable. **SortaGrad** is a simple curriculum: in the first epoch, mini-batches are taken in order of increasing utterance length, and later epochs are random. Long utterances have larger CTC losses and gradients early in training, so starting short is more stable. For streaming, a unidirectional model adds a small **lookahead convolution** over a few future frames instead of the backward RNN. Training used 11,940 h of English and 9,400 h of Mandarin.

**Reference numbers**

- **Deep Speech:** 5 layers (3 FC, 1 bidirectional RNN, 1 FC); clipped ReLU at 20; characters + blank; about 5,000 h of in-house read speech; 16.0% WER on Hub5'00 (full)
- **Deep Speech 2:** up to 3 conv + 7 recurrent layers; sequence-wise BatchNorm; SortaGrad; 11,940 h English and 9,400 h Mandarin
- **DS2 on LibriSpeech:** 5.33% WER test-clean, 13.25% test-other, close to the human transcribers measured in the same paper (5.83% and 12.69%)
- **Decoder:** CTC prefix beam search with an n-gram LM: $Q=\log P_{\text{ctc}}+\alpha\log P_{\text{lm}}+\beta\,\text{words}$; $\alpha,\beta$ tuned on a dev set

## Decoding with a language model

A character CTC model spells phonetically: on its own it writes "bostin" or "cabtoboston" when the sound is ambiguous. Both papers therefore decode with a beam search over character prefixes (prefix beam search; see **Connectionist Temporal Classification** and **Beam search**). An n-gram language model is consulted each time a word is completed.

$$
Q(c)=\log P_{\text{ctc}}(c\mid x)+\alpha\,\log P_{\text{lm}}(c)+\beta\,\mathrm{word\_count}(c)
$$

- $P_{\text{ctc}}(c\mid x)$: sum over all frame paths that collapse to the character string $c$, kept per prefix as blank-ending and label-ending parts
- $\alpha$: language-model weight: how much to trust the text prior over the acoustics
- $\beta$: word insertion bonus. Every word multiplies in an LM probability below 1, so without $\beta$ the search prefers fewer words

> Figure: A 21-frame CTC output for "a cab to boston" with three planted ambiguities: a weak "a" (0.50 vs blank 0.42), a noise blip after "cab" (a = 0.2), and "i" (0.52) beating "o" (0.38) in "boston". Greedy decoding gives "a cab to bostin". At the default α = 0.8, β = 1, beam 8, the winner is the reference, with Q = −4.29 − 4.50 + 4.00 = −4.79; "cab to boston" is second at −5.99. With α = 0 the acoustics win and the result is "bostin" (Q = −4.00 at β = 0). With α = 1.5 and β = 0 the LM deletes the weak "a", giving "cab to boston". At α = 0.8 and β = 5 the bonus inserts a word: "a cab a to boston". At α = 0 and β = 4 the result is the nonsense "a c ab a to b st n".

Worked example at α = 0.8, β = 1 (natural logs; bigram LM: p(a|⟨s⟩) = 0.3, p(cab|a) = 0.1, p(cab|⟨s⟩) = 0.1, p(to|cab) = 0.4, p(boston|to) = 0.3, out-of-vocabulary words backed off to 0.4 × 10⁻⁴)

| Hypothesis | log P_ctc | log P_lm | α·log P_lm | β·words | Q |
| --- | --- | --- | --- | --- | --- |
| a cab to boston | −4.29 | −1.20 − 2.30 − 0.92 − 1.20 = −5.63 | −4.50 | +4.00 | −4.79 |
| cab to boston | −5.45 | −2.30 − 0.92 − 1.20 = −4.42 | −3.54 | +3.00 | −5.99 |
| a cab to bostin | −4.00 | −1.20 − 2.30 − 0.92 − 10.13 = −14.55 | −11.64 | +4.00 | −11.64 |

"bostin" has the best acoustic score but is out of vocabulary, and its $\ln(0.4\times10^{-4})=-10.13$ sinks it. Dropping "a" saves $0.8\times1.20=0.96$ of LM cost. However, it loses $1.16$ of acoustic score and the $\beta=1$ word bonus, so the reference wins by $1.20$.

```python
import soundfile as sf
from deepspeech import Model

ds = Model("deepspeech-0.9.3-models.pbmm")            # acoustic model (TensorFlow graph)
ds.enableExternalScorer("deepspeech-0.9.3-models.scorer")  # KenLM n-gram + vocabulary trie
ds.setScorerAlphaBeta(0.93, 1.18)                     # alpha = LM weight, beta = word bonus
ds.setBeamWidth(500)

audio, sr = sf.read("cab.wav", dtype="int16")         # 16 kHz mono PCM
assert sr == ds.sampleRate()
print(ds.stt(audio))

ds.disableExternalScorer()                            # acoustic-only decoding, for comparison
print(ds.stt(audio))
```

Mozilla DeepSpeech inference: acoustic model plus KenLM scorer, with α and β set explicitly (not run here; the deepspeech package is archived and needs an old Python)

> **α and β pull in opposite directions:** Raising $\alpha$ fixes spellings toward real words, but it also charges every word an LM factor, so the search starts dropping short, weakly heard words (deletions). $\beta$ pays a fixed reward per word to cancel that bias; too much of it and the search invents words (insertions). They are therefore tuned together, usually by grid search on a dev set, and they must be retuned whenever the LM, its vocabulary or the acoustic model changes.

> **Mozilla DeepSpeech:** Mozilla's open-source engine (from 2017) followed the 2014 paper's design. It replaced the bidirectional RNN with a unidirectional LSTM so that it could stream, and it read MFCC features. A KenLM scorer applied the $\alpha,\beta$ formula above. It ran offline on desktops and small devices. Development later stopped, and the Coqui STT fork continued it.

**Strengths**

- One network from spectrogram to characters: no lexicon, phone set or forced alignment
- CTC is non-autoregressive; DS2's lookahead variant streams
- An external LM can be swapped in for a new domain without retraining the acoustic model

**Limitations**

- CTC frames are conditionally independent, so good spelling depends on the external LM
- α and β need tuning and interact with beam width and vocabulary
- Needs thousands of labelled hours; later self-supervised and attention models need far less labelled data

**Question:** You raise α and the decoder starts dropping short words such as "a" and "the". Which change targets this directly?

- A. Increase the beam width
- B. Increase β, the word insertion bonus
- C. Lower the clipped-ReLU ceiling
- D. Remove the blank symbol

**Answer:** B. Increase β, the word insertion bonus — Each extra word multiplies in an LM probability below 1, which costs $\alpha\log p<0$. $\beta$ adds a fixed reward per word to offset that. A wider beam only helps when the right answer was pruned; here the score itself prefers the shorter output.

**Question:** What is SortaGrad?

- A. Sorting the vocabulary by frequency before training
- B. Clipping gradients by their sorted magnitude
- C. Presenting utterances in order of increasing length during the first epoch, then shuffling
- D. Sorting beam hypotheses by length during decoding

**Answer:** C. Presenting utterances in order of increasing length during the first epoch, then shuffling — Long utterances have larger CTC losses and gradients and destabilise early training. Starting with short ones is a curriculum. Deep Speech 2 found that it lowered final error and made deep recurrent models train more reliably.

**Key points**

- Deep Speech (2014): spectrogram → 3 clipped-ReLU FC layers → bidirectional RNN → FC → characters, trained with CTC.
- Scale and synthesis: about 5,000 h of in-house speech, noise superposition and Lombard-effect recordings; 16.0% WER on Hub5'00.
- Deep Speech 2 (2016): conv front end, up to 7 bidirectional GRU/RNN layers, sequence-wise BatchNorm, SortaGrad; English and Mandarin.
- Decode: $Q=\log P_{\text{ctc}}+\alpha\log P_{\text{lm}}+\beta\cdot\text{words}$; $\alpha$ fixes spelling, $\beta$ offsets the LM's bias toward fewer words.
- Mozilla DeepSpeech made the design open source: a streaming LSTM variant with a KenLM scorer.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
