Definition
Speech-to-text (automatic speech recognition, ASR) maps an audio waveform to the word sequence that was spoken, . Modern systems are neural networks trained end to end on (audio, transcript) pairs. The main families are CTC models, attention encoder–decoders such as Listen, Attend and Spell (Chan et al., 2016) and Whisper (Radford et al., 2022), and RNN transducers (Graves, 2012). Self-supervised encoders such as wav2vec 2.0 (Baevski et al., 2020) are often used underneath.
The input is long and the output is short. Ten seconds of 16 kHz audio is 160,000 samples, or 1,000 feature frames, but the transcript may be 30 characters. Nobody marks where each word starts. The same word also sounds different across speakers, accents, microphones and noise.
Before deep learning, recognisers chained separate parts. Gaussian mixture HMMs modelled MFCC features, a pronunciation lexicon mapped phones to words, and an n-gram language model scored word sequences. Around 2012, hybrid DNN–HMM systems (Hinton et al., 2012) replaced the Gaussians with a neural frame classifier. End-to-end models then replaced the lexicon and the alignment model as well.
Front end: the log-mel spectrogram
Most models do not read raw samples. A short-time Fourier transform is taken over 25 ms windows every 10 ms. At 16 kHz that is 400-sample windows with a 160-sample hop, or 100 frames per second. The power spectrum is pooled by triangular filters equally spaced on the mel scale, usually 80 of them, and then a log is applied.
- mel value of frequency in Hz. Low frequencies get finer resolution, like human pitch perception ()
- weight of FFT bin in triangular mel filter
- log-mel feature at frame , band : the network's input "image"
Three ways to decode
CTC. The encoder emits a softmax over the vocabulary plus a blank at every frame. A path collapses to a label sequence by merging repeats and then deleting blanks. Training sums over all paths that collapse to the transcript. This is covered in Connectionist Temporal Classification and CTC loss. CTC is fast and non-autoregressive, but it assumes frames are independent given the audio. It therefore benefits from an external language model during Beam search.
- encoder frames; the decoder reads them through Cross-attention and models the language implicitly
- transducer encoder output at frame , and prediction-network state after emitted labels
- joint network over vocabulary + blank. Emitting blank moves to frame ; emitting a label moves to
The attention encoder–decoder is the most accurate when it can see the whole utterance. However, it can loop, skip words or invent text, because nothing forces a monotonic alignment. The RNN-T keeps CTC's monotonic, frame-synchronous search and adds label context. It can stream frame by frame, which is why it is common in on-device dictation. Hybrids train a shared encoder with CTC and attention losses together.
Pre-training: wav2vec 2.0 and Whisper
wav2vec 2.0 (Baevski, Zhou, Mohamed and Auli, 2020) learns from unlabelled audio. A CNN reads the raw waveform and emits one vector per 20 ms (stride 320 samples). Spans of these vectors are masked, and a Transformer must pick the true quantised vector for each masked step from distractors with a contrastive loss (see InfoNCE loss). Fine-tuned with CTC on LibriSpeech, the large model reached 1.8/3.3 WER on test-clean/test-other with all 960 h of labels, and 4.8/8.2 with only 10 minutes.
Whisper (Radford et al., 2022, "Robust Speech Recognition via Large-Scale Weak Supervision") went the other way: plain supervised training on 680,000 hours of weakly labelled web audio. It covers many languages and includes X→English translation. It is an encoder–decoder Transformer over 30 s windows of 80-bin log-mel. Special tokens in the decoder prompt choose the task: language tag, transcribe or translate, with or without timestamps.
Reference points
- LibriSpeech
- about 1,000 h of read English audiobooks (Panayotov et al., 2015); 960 h of training data; test-clean and the harder test-other
- Features
- 16 kHz, 25 ms / 10 ms, 80 log-mel bins → 100 frames/s; Whisper large-v3 uses 128 bins
- Whisper sizes
- tiny 39M, base 74M, small 244M, medium 769M, large 1550M parameters (large: 32 layers, width 1280)
- wav2vec 2.0
- BASE 95M and LARGE 317M parameters; 50 encoder frames per second
- Metrics
- WER for words; CER for characters (better for languages written without spaces)
Word error rate
- substitutions, deletions and insertions in a minimum-edit (Levenshtein) alignment of hypothesis words to reference words
- number of reference words. Insertions are not capped, so WER can exceed 100%
| Reference | Hypothesis | Operation | Running edits |
|---|---|---|---|
| · | so | insertion | 1 |
| set | set | match | 1 |
| a | the | substitution | 2 |
| timer | timer | match | 2 |
| for | for | match | 2 |
| ten | ten | match | 2 |
| minutes | minutes | match | 2 |
| please | · | deletion | 3 |
So , , , and WER . Both strings have 7 words, so naively comparing word by word would give 6 mismatches of 7. The alignment step is essential.
Shapes for one 30 s Whisper window
- 1 samples; window 400, hop 160 → frames.
- 2Log-mel input: numbers.
- 3Two convolutions with filter width 3, the second with stride 2 → 1500 encoder positions of 20 ms each.
- 4wav2vec 2.0 on the same clip: its 7-layer CNN (total stride 320) gives 1499 frames, about 50 per second.
1import re2from transformers import pipeline34asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")5hyp = asr("timer.wav")["text"] # 16 kHz audio is resampled/padded to 30 s internally67def normalise(s):8 return re.sub(r"[^\w' ]", "", s.lower()).split()910def wer(ref, hyp):11 r, h = normalise(ref), normalise(hyp)12 d = list(range(len(h) + 1)) # one row of the Levenshtein table13 for i in range(1, len(r) + 1):14 prev, d[0] = d[0], i15 for j in range(1, len(h) + 1):16 prev, d[j] = d[j], min(prev + (r[i-1] != h[j-1]), d[j] + 1, d[j-1] + 1)17 return d[-1] / len(r)1819ref = "set a timer for ten minutes please"20print(wer(ref, "so set the timer for ten minutes")) # 0.428621print(wer(ref, "Set a timer for 10 minutes, please.")) # 0.1429 ("10" vs "ten")Use when
- +CTC: parallel, fast and monotonic; pairs well with self-supervised encoders
- +Attention encoder–decoder: best accuracy offline; one model can handle language ID, translation and timestamps
- +RNN-T: streaming with label context; the standard for on-device dictation
Avoid when
- −CTC: conditionally independent frames; needs an LM for good spelling
- −Attention: needs the whole window, autoregressive, can hallucinate or loop
- −RNN-T: memory-heavy T × U joint lattice in training
Check yourself
Reference has 5 words. The hypothesis contains all 5, correctly, plus 6 extra words. What is the WER?
Check yourself
Why does an RNN-T stream more naturally than an attention encoder–decoder?
Remember
- ✓Input: 80-bin log-mel, 25 ms window, 10 ms hop → 100 frames/s; mel .
- ✓Decoders: CTC (frame-wise plus collapse), attention encoder–decoder (LAS, Whisper), RNN-T (streaming).
- ✓wav2vec 2.0: contrastive self-supervision on raw audio; 4.8/8.2 LibriSpeech WER from 10 minutes of labels.
- ✓Whisper: 680k hours of weak supervision, 30 s windows, task chosen by special tokens.
- ✓WER from a Levenshtein alignment; normalise text first; it can exceed 100%.