Speech-to-text

Turn audio into words: log-mel frames (25 ms window, 10 ms hop) go through a neural encoder, then a CTC, attention or transducer decoder. Systems are scored by WER = (S+D+I)/N.

8 min read778 wordscore0 viewsupdated 2026-10-15

Definition

Speech-to-text (automatic speech recognition, ASR) maps an audio waveform xx to the word sequence yy that was spoken, y^=arg⁡max⁡yp(y∣x)\hat y=\arg\max_y p(y\mid x). Modern systems are neural networks trained end to end on (audio, transcript) pairs. The main families are CTC models, attention encoder–decoders such as Listen, Attend and Spell (Chan et al., 2016) and Whisper (Radford et al., 2022), and RNN transducers (Graves, 2012). Self-supervised encoders such as wav2vec 2.0 (Baevski et al., 2020) are often used underneath.

The input is long and the output is short. Ten seconds of 16 kHz audio is 160,000 samples, or 1,000 feature frames, but the transcript may be 30 characters. Nobody marks where each word starts. The same word also sounds different across speakers, accents, microphones and noise.

Before deep learning, recognisers chained separate parts. Gaussian mixture HMMs modelled MFCC features, a pronunciation lexicon mapped phones to words, and an n-gram language model scored word sequences. Around 2012, hybrid DNN–HMM systems (Hinton et al., 2012) replaced the Gaussians with a neural frame classifier. End-to-end models then replaced the lexicon and the alignment model as well.

Front end: the log-mel spectrogram

Most models do not read raw samples. A short-time Fourier transform is taken over 25 ms windows every 10 ms. At 16 kHz that is 400-sample windows with a 160-sample hop, or 100 frames per second. The power spectrum is pooled by triangular filters equally spaced on the mel scale, usually 80 of them, and then a log is applied.

m(f)=2595 log⁡10 ⁣(1+f700),Xt,k=log⁡(ε+∑jHk(j) ∣STFT(x)t,j∣2)m(f)=2595\,\log_{10}\!\Big(1+\frac{f}{700}\Big),\qquad X_{t,k}=\log\Big(\varepsilon+\sum_{j} H_k(j)\,\big|\mathrm{STFT}(x)_{t,j}\big|^2\Big)
m(f)m(f)
mel value of frequency ff in Hz. Low frequencies get finer resolution, like human pitch perception (m(1000)≈1000m(1000)\approx1000)
Hk(j)H_k(j)
weight of FFT bin jj in triangular mel filter k=1,…,80k=1,\dots,80
Xt,kX_{t,k}
log-mel feature at frame tt, band kk: the network's input "image"
front end (fixed signal processing) · Whisper-style numbers for a 30 s window at 16 kHz480,000 sampleswaveform 16 kHzframe + STFT25 ms window= 400 samples10 ms hop = 160→ 3000 framesmel + log80 triangularfilters on themel scalelog10, rescalelog-mel 80 × 3000acoustic encoder: 2 conv (GELU, 2nd stride 2) → 1500 frames × d, 20 ms eachthen Transformer or Conformer blocks · or wav2vec 2.0: CNN on raw audio → 50 frames/s,pre-trained self-supervised on unlabelled speech, fine-tuned with CTCCTCsoftmax per frameover V + blank "-"ss-e-tt__a → set_amerge repeats, drop -non-autoregressiveframes independentattention enc–decdecoder cross-attendsto all 1500 frames<|en|><|transcribe|>→ set a timer …LAS, Whisperneeds full utteranceRNN-T (transducer)joint(f_t, g_u) → V + -f_t: encoder frameg_u: label history- → advance tstreaming, on-devicemodels label context“set a timer for ten minutes please” · scored by WER = (S + D + I) / N
fig.The ASR pipeline with Whisper's numbers: a 30 s, 16 kHz clip becomes an 80 × 3000 log-mel spectrogram. A convolutional stem with stride 2 reduces it to 1500 encoder frames of 20 ms each. One of three decoders then reads those frames: CTC (per-frame softmax plus collapse; _ marks a space), an attention decoder that cross-attends to all frames, or an RNN transducer that combines encoder frames with label history. Output is scored by WER.

Three ways to decode

CTC. The encoder emits a softmax over the vocabulary plus a blank at every frame. A path collapses to a label sequence by merging repeats and then deleting blanks. Training sums over all paths that collapse to the transcript. This is covered in Connectionist Temporal Classification and CTC loss. CTC is fast and non-autoregressive, but it assumes frames are independent given the audio. It therefore benefits from an external language model during Beam search.

attention:   p(y∣x)=∏u=1Up(yu∣y<u, Enc(x))RNN-T:   p(k∣t,u)=softmax(J(ft+gu))\text{attention: }\;p(y\mid x)=\prod_{u=1}^{U}p\big(y_u\mid y_{<u},\,\mathrm{Enc}(x)\big)\qquad\text{RNN-T: }\;p(k\mid t,u)=\mathrm{softmax}\big(J(f_t+g_u)\big)
Enc(x)\mathrm{Enc}(x)
encoder frames; the decoder reads them through Cross-attention and models the language implicitly
ft,  guf_t,\;g_u
transducer encoder output at frame tt, and prediction-network state after uu emitted labels
JJ
joint network over vocabulary + blank. Emitting blank moves to frame t+1t+1; emitting a label moves to u+1u+1

The attention encoder–decoder is the most accurate when it can see the whole utterance. However, it can loop, skip words or invent text, because nothing forces a monotonic alignment. The RNN-T keeps CTC's monotonic, frame-synchronous search and adds label context. It can stream frame by frame, which is why it is common in on-device dictation. Hybrids train a shared encoder with CTC and attention losses together.

Pre-training: wav2vec 2.0 and Whisper

wav2vec 2.0 (Baevski, Zhou, Mohamed and Auli, 2020) learns from unlabelled audio. A CNN reads the raw waveform and emits one vector per 20 ms (stride 320 samples). Spans of these vectors are masked, and a Transformer must pick the true quantised vector for each masked step from distractors with a contrastive loss (see InfoNCE loss). Fine-tuned with CTC on LibriSpeech, the large model reached 1.8/3.3 WER on test-clean/test-other with all 960 h of labels, and 4.8/8.2 with only 10 minutes.

Whisper (Radford et al., 2022, "Robust Speech Recognition via Large-Scale Weak Supervision") went the other way: plain supervised training on 680,000 hours of weakly labelled web audio. It covers many languages and includes X→English translation. It is an encoder–decoder Transformer over 30 s windows of 80-bin log-mel. Special tokens in the decoder prompt choose the task: language tag, transcribe or translate, with or without timestamps.

Reference points

LibriSpeech
about 1,000 h of read English audiobooks (Panayotov et al., 2015); 960 h of training data; test-clean and the harder test-other
Features
16 kHz, 25 ms / 10 ms, 80 log-mel bins → 100 frames/s; Whisper large-v3 uses 128 bins
Whisper sizes
tiny 39M, base 74M, small 244M, medium 769M, large 1550M parameters (large: 32 layers, width 1280)
wav2vec 2.0
BASE 95M and LARGE 317M parameters; 50 encoder frames per second
Metrics
WER for words; CER for characters (better for languages written without spaces)

Word error rate

WER=S+D+IN\mathrm{WER}=\frac{S+D+I}{N}
S, D, IS,\,D,\,I
substitutions, deletions and insertions in a minimum-edit (Levenshtein) alignment of hypothesis words to reference words
NN
number of reference words. Insertions are not capped, so WER can exceed 100%
alignment · reference N = 7 words · hypothesis 7 words · normaliser: lowercase + strip punct.refhyp·soinssetsetmatchathesubtimertimermatchforformatchtentenmatchminutesminutesmatchplease·delD[i][j] = min(D[i−1][j−1] + [ref_i ≠ hyp_j], D[i−1][j] + 1, D[i][j−1] + 1)εsosetthetimerfortenminutesεsetatimerfortenminutesplease0123456711123456222234563333234544443234555543236666543277776543rows: reference words · columns: hypothesis words · coloured cells: one optimal backtrace (↘ match/sub, ↓ del, → ins)S = 1 · D = 1 · I = 1 → WER = (1+1+1)/7 = 42.9%CER (characters incl. spaces) = 13/34 = 38.2% · word accuracy 1 − WER = 57.1%same edit count, any mix: 3 edits over 7 reference words
hypothesis
normaliser
“so set the timer for ten minutes”
fig.The reference is "set a timer for ten minutes please" (N = 7). The default S + D + I hypothesis has one insertion (so), one substitution (a → the) and one deletion (please), so WER = 3/7 = 42.9% and CER = 38.2%. The sound-alikes hypothesis has the same 42.9% WER but only 17.6% CER: every near-miss still costs a whole word. Under the raw normaliser the formatted hypothesis scores 57.1% (4 substitutions). Lowercasing and stripping punctuation drops it to 14.3%, and spelling out numbers drops it to 0%. The hallucinated tail adds 4 insertions, giving 57.1%.
ReferenceHypothesisOperationRunning edits
·soinsertion1
setsetmatch1
athesubstitution2
timertimermatch2
forformatch2
tentenmatch2
minutesminutesmatch2
please·deletion3
Worked example: reference "set a timer for ten minutes please" vs hypothesis "so set the timer for ten minutes"

So S=1S=1, D=1D=1, I=1I=1, N=7N=7 and WER =3/7=42.9%=3/7=42.9\%. Both strings have 7 words, so naively comparing word by word would give 6 mismatches of 7. The alignment step is essential.

Shapes for one 30 s Whisper window

  1. 130 s×16,000=480,00030\text{ s}\times16{,}000=480{,}000 samples; window 400, hop 160 → 480,000/160=3000480{,}000/160=3000 frames.
  2. 2Log-mel input: 80×3000=240,00080\times3000=240{,}000 numbers.
  3. 3Two convolutions with filter width 3, the second with stride 2 → 1500 encoder positions of 20 ms each.
  4. 4wav2vec 2.0 on the same clip: its 7-layer CNN (total stride 320) gives 1499 frames, about 50 per second.
python
1import re
2from transformers import pipeline
3
4asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
5hyp = asr("timer.wav")["text"] # 16 kHz audio is resampled/padded to 30 s internally
6
7def normalise(s):
8 return re.sub(r"[^\w' ]", "", s.lower()).split()
9
10def wer(ref, hyp):
11 r, h = normalise(ref), normalise(hyp)
12 d = list(range(len(h) + 1)) # one row of the Levenshtein table
13 for i in range(1, len(r) + 1):
14 prev, d[0] = d[0], i
15 for j in range(1, len(h) + 1):
16 prev, d[j] = d[j], min(prev + (r[i-1] != h[j-1]), d[j] + 1, d[j-1] + 1)
17 return d[-1] / len(r)
18
19ref = "set a timer for ten minutes please"
20print(wer(ref, "so set the timer for ten minutes")) # 0.4286
21print(wer(ref, "Set a timer for 10 minutes, please.")) # 0.1429 ("10" vs "ten")
Transcribe with Whisper (Hugging Face), then score with a small WER function

Use when

  • +CTC: parallel, fast and monotonic; pairs well with self-supervised encoders
  • +Attention encoder–decoder: best accuracy offline; one model can handle language ID, translation and timestamps
  • +RNN-T: streaming with label context; the standard for on-device dictation

Avoid when

  • −CTC: conditionally independent frames; needs an LM for good spelling
  • −Attention: needs the whole window, autoregressive, can hallucinate or loop
  • −RNN-T: memory-heavy T × U joint lattice in training

Check yourself

Reference has 5 words. The hypothesis contains all 5, correctly, plus 6 extra words. What is the WER?

Check yourself

Why does an RNN-T stream more naturally than an attention encoder–decoder?

Remember

  • ✓Input: 80-bin log-mel, 25 ms window, 10 ms hop → 100 frames/s; mel m(f)=2595log⁡10(1+f/700)m(f)=2595\log_{10}(1+f/700).
  • ✓Decoders: CTC (frame-wise plus collapse), attention encoder–decoder (LAS, Whisper), RNN-T (streaming).
  • ✓wav2vec 2.0: contrastive self-supervision on raw audio; 4.8/8.2 LibriSpeech WER from 10 minutes of labels.
  • ✓Whisper: 680k hours of weak supervision, 30 s windows, task chosen by special tokens.
  • ✓WER =(S+D+I)/N=(S+D+I)/N from a Levenshtein alignment; normalise text first; it can exceed 100%.
← back to Deep Learning roadmap