Definition
DeepSpeech names two Baidu Research systems and an open-source descendant. Deep Speech (Hannun, Case, Casper, Catanzaro, Diamos, Elsen, Prenger, Satheesh, Sengupta, Coates and Ng, 2014, "Deep Speech: Scaling up end-to-end speech recognition") is a 5-layer RNN trained with CTC from spectrograms straight to characters. Deep Speech 2 (Amodei et al., ICML 2016, "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin") is a deeper convolutional-recurrent version. Mozilla DeepSpeech is an open-source TensorFlow engine based on the first paper.
In 2014 the best recognisers were hybrid pipelines. A DNN classified frames into HMM states. Those state targets came from a forced alignment made by an earlier GMM system. A pronunciation lexicon mapped phones to words, and speaker adaptation and hand-designed features sat on top. Each part was trained separately.
Deep Speech replaced the acoustic model, the lexicon and the alignments with one network. It reads spectrogram frames and outputs a character distribution per frame, trained with Connectionist Temporal Classification, so no frame labels or phonemes are needed. Its other bet was scale: thousands of hours of speech, multi-GPU training, and synthetic noise to make the model robust.
Deep Speech (2014): five layers and CTC
The input at time is a spectrogram frame together with neighbouring frames on each side (). Three fully connected layers come first. They use a clipped ReLU, , which keeps activations bounded in the recurrent layer. Layer 4 is a bidirectional recurrent layer. It is a plain RNN rather than an LSTM, chosen because it is cheaper to compute and to store. Layer 5 is fully connected, followed by a softmax over the characters a–z, space, apostrophe and the CTC blank.
- output of the third fully connected layer at frame
- forward recurrence (left to right) and backward recurrence (right to left); they are summed, not concatenated
- clipped ReLU
The training data were about 5,000 h of read speech from 9,600 speakers, collected by Baidu, plus Switchboard, Fisher and WSJ. To make the model robust to noise, clean utterances were mixed with many noise tracks. Some speakers were also recorded while hearing loud noise in headphones, which captures the Lombard effect: people raise and change their voice in noise. On the full Switchboard Hub5'00 test set, a model trained on the 2,300 h of Switchboard and Fisher reached 16.0% WER.
Deep Speech 2 (2016): deeper, faster, two languages
Deep Speech 2 kept CTC and the character outputs but rebuilt the network. One to three 2D convolutions over time and frequency come first, with striding in time to shorten the sequence. Then come up to seven bidirectional recurrent layers (see Gated recurrent unit (GRU)). Batch normalization is applied only to the input-to-hidden connections and normalised over the whole sequence. The same architecture works for English and Mandarin; for Mandarin it outputs Chinese characters directly.
Two training tricks made the deep model trainable. SortaGrad is a simple curriculum: in the first epoch, mini-batches are taken in order of increasing utterance length, and later epochs are random. Long utterances have larger CTC losses and gradients early in training, so starting short is more stable. For streaming, a unidirectional model adds a small lookahead convolution over a few future frames instead of the backward RNN. Training used 11,940 h of English and 9,400 h of Mandarin.
Reference numbers
- Deep Speech
- 5 layers (3 FC, 1 bidirectional RNN, 1 FC); clipped ReLU at 20; characters + blank; about 5,000 h of in-house read speech; 16.0% WER on Hub5'00 (full)
- Deep Speech 2
- up to 3 conv + 7 recurrent layers; sequence-wise BatchNorm; SortaGrad; 11,940 h English and 9,400 h Mandarin
- DS2 on LibriSpeech
- 5.33% WER test-clean, 13.25% test-other, close to the human transcribers measured in the same paper (5.83% and 12.69%)
- Decoder
- CTC prefix beam search with an n-gram LM: ; tuned on a dev set
Decoding with a language model
A character CTC model spells phonetically: on its own it writes "bostin" or "cabtoboston" when the sound is ambiguous. Both papers therefore decode with a beam search over character prefixes (prefix beam search; see Connectionist Temporal Classification and Beam search). An n-gram language model is consulted each time a word is completed.
- sum over all frame paths that collapse to the character string , kept per prefix as blank-ending and label-ending parts
- language-model weight: how much to trust the text prior over the acoustics
- word insertion bonus. Every word multiplies in an LM probability below 1, so without the search prefers fewer words
| Hypothesis | log P_ctc | log P_lm | α·log P_lm | β·words | Q |
|---|---|---|---|---|---|
| a cab to boston | −4.29 | −1.20 − 2.30 − 0.92 − 1.20 = −5.63 | −4.50 | +4.00 | −4.79 |
| cab to boston | −5.45 | −2.30 − 0.92 − 1.20 = −4.42 | −3.54 | +3.00 | −5.99 |
| a cab to bostin | −4.00 | −1.20 − 2.30 − 0.92 − 10.13 = −14.55 | −11.64 | +4.00 | −11.64 |
"bostin" has the best acoustic score but is out of vocabulary, and its sinks it. Dropping "a" saves of LM cost. However, it loses of acoustic score and the word bonus, so the reference wins by .
1import soundfile as sf2from deepspeech import Model34ds = Model("deepspeech-0.9.3-models.pbmm") # acoustic model (TensorFlow graph)5ds.enableExternalScorer("deepspeech-0.9.3-models.scorer") # KenLM n-gram + vocabulary trie6ds.setScorerAlphaBeta(0.93, 1.18) # alpha = LM weight, beta = word bonus7ds.setBeamWidth(500)89audio, sr = sf.read("cab.wav", dtype="int16") # 16 kHz mono PCM10assert sr == ds.sampleRate()11print(ds.stt(audio))1213ds.disableExternalScorer() # acoustic-only decoding, for comparison14print(ds.stt(audio))Use when
- +One network from spectrogram to characters: no lexicon, phone set or forced alignment
- +CTC is non-autoregressive; DS2's lookahead variant streams
- +An external LM can be swapped in for a new domain without retraining the acoustic model
Avoid when
- −CTC frames are conditionally independent, so good spelling depends on the external LM
- −α and β need tuning and interact with beam width and vocabulary
- −Needs thousands of labelled hours; later self-supervised and attention models need far less labelled data
Check yourself
You raise α and the decoder starts dropping short words such as "a" and "the". Which change targets this directly?
Check yourself
What is SortaGrad?
Remember
- ✓Deep Speech (2014): spectrogram → 3 clipped-ReLU FC layers → bidirectional RNN → FC → characters, trained with CTC.
- ✓Scale and synthesis: about 5,000 h of in-house speech, noise superposition and Lombard-effect recordings; 16.0% WER on Hub5'00.
- ✓Deep Speech 2 (2016): conv front end, up to 7 bidirectional GRU/RNN layers, sequence-wise BatchNorm, SortaGrad; English and Mandarin.
- ✓Decode: ; fixes spelling, offsets the LM's bias toward fewer words.
- ✓Mozilla DeepSpeech made the design open source: a streaming LSTM variant with a KenLM scorer.