# Speech-to-text

> Turn audio into words: log-mel frames (25 ms window, 10 ms hop) go through a neural encoder, then a CTC, attention or transducer decoder. Systems are scored by WER = (S+D+I)/N.

Source: https://www.mlfoundry.in/learn/speech-to-text  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**Speech-to-text (automatic speech recognition, ASR):** maps an audio waveform $x$ to the word sequence $y$ that was spoken, $\hat y=\arg\max_y p(y\mid x)$. Modern systems are neural networks trained end to end on (audio, transcript) pairs. The main families are CTC models, attention encoder–decoders such as Listen, Attend and Spell (Chan et al., 2016) and Whisper (Radford et al., 2022), and RNN transducers (Graves, 2012). Self-supervised encoders such as wav2vec 2.0 (Baevski et al., 2020) are often used underneath.

The input is long and the output is short. Ten seconds of 16 kHz audio is 160,000 samples, or 1,000 feature frames, but the transcript may be 30 characters. Nobody marks where each word starts. The same word also sounds different across speakers, accents, microphones and noise.

Before deep learning, recognisers chained separate parts. Gaussian mixture HMMs modelled MFCC features, a pronunciation lexicon mapped phones to words, and an n-gram language model scored word sequences. Around 2012, hybrid DNN–HMM systems (Hinton et al., 2012) replaced the Gaussians with a neural frame classifier. End-to-end models then replaced the lexicon and the alignment model as well.

## Front end: the log-mel spectrogram

Most models do not read raw samples. A short-time Fourier transform is taken over **25 ms windows every 10 ms**. At 16 kHz that is 400-sample windows with a 160-sample hop, or 100 frames per second. The power spectrum is pooled by triangular filters equally spaced on the mel scale, usually **80** of them, and then a log is applied.

$$
m(f)=2595\,\log_{10}\!\Big(1+\frac{f}{700}\Big),\qquad X_{t,k}=\log\Big(\varepsilon+\sum_{j} H_k(j)\,\big|\mathrm{STFT}(x)_{t,j}\big|^2\Big)
$$

- $m(f)$: mel value of frequency $f$ in Hz. Low frequencies get finer resolution, like human pitch perception ($m(1000)\approx1000$)
- $H_k(j)$: weight of FFT bin $j$ in triangular mel filter $k=1,\dots,80$
- $X_{t,k}$: log-mel feature at frame $t$, band $k$: the network's input "image"

> Figure: The ASR pipeline with Whisper's numbers: a 30 s, 16 kHz clip becomes an 80 × 3000 log-mel spectrogram. A convolutional stem with stride 2 reduces it to 1500 encoder frames of 20 ms each. One of three decoders then reads those frames: CTC (per-frame softmax plus collapse; _ marks a space), an attention decoder that cross-attends to all frames, or an RNN transducer that combines encoder frames with label history. Output is scored by WER.

## Three ways to decode

**CTC.** The encoder emits a softmax over the vocabulary plus a blank at every frame. A path collapses to a label sequence by merging repeats and then deleting blanks. Training sums over all paths that collapse to the transcript. This is covered in **Connectionist Temporal Classification** and **CTC loss**. CTC is fast and non-autoregressive, but it assumes frames are independent given the audio. It therefore benefits from an external language model during **Beam search**.

$$
\text{attention: }\;p(y\mid x)=\prod_{u=1}^{U}p\big(y_u\mid y_{<u},\,\mathrm{Enc}(x)\big)\qquad\text{RNN-T: }\;p(k\mid t,u)=\mathrm{softmax}\big(J(f_t+g_u)\big)
$$

- $\mathrm{Enc}(x)$: encoder frames; the decoder reads them through **Cross-attention** and models the language implicitly
- $f_t,\;g_u$: transducer encoder output at frame $t$, and prediction-network state after $u$ emitted labels
- $J$: joint network over vocabulary + blank. Emitting blank moves to frame $t+1$; emitting a label moves to $u+1$

The attention encoder–decoder is the most accurate when it can see the whole utterance. However, it can loop, skip words or invent text, because nothing forces a monotonic alignment. The RNN-T keeps CTC's monotonic, frame-synchronous search and adds label context. It can stream frame by frame, which is why it is common in on-device dictation. Hybrids train a shared encoder with CTC and attention losses together.

## Pre-training: wav2vec 2.0 and Whisper

**wav2vec 2.0** (Baevski, Zhou, Mohamed and Auli, 2020) learns from unlabelled audio. A CNN reads the raw waveform and emits one vector per 20 ms (stride 320 samples). Spans of these vectors are masked, and a Transformer must pick the true quantised vector for each masked step from distractors with a contrastive loss (see **InfoNCE loss**). Fine-tuned with CTC on LibriSpeech, the large model reached 1.8/3.3 WER on test-clean/test-other with all 960 h of labels, and 4.8/8.2 with only 10 minutes.

**Whisper** (Radford et al., 2022, "Robust Speech Recognition via Large-Scale Weak Supervision") went the other way: plain supervised training on **680,000 hours** of weakly labelled web audio. It covers many languages and includes X→English translation. It is an encoder–decoder Transformer over **30 s windows** of 80-bin log-mel. Special tokens in the decoder prompt choose the task: language tag, transcribe or translate, with or without timestamps.

**Reference points**

- **LibriSpeech:** about 1,000 h of read English audiobooks (Panayotov et al., 2015); 960 h of training data; test-clean and the harder test-other
- **Features:** 16 kHz, 25 ms / 10 ms, 80 log-mel bins → 100 frames/s; Whisper large-v3 uses 128 bins
- **Whisper sizes:** tiny 39M, base 74M, small 244M, medium 769M, large 1550M parameters (large: 32 layers, width 1280)
- **wav2vec 2.0:** BASE 95M and LARGE 317M parameters; 50 encoder frames per second
- **Metrics:** WER for words; CER for characters (better for languages written without spaces)

## Word error rate

$$
\mathrm{WER}=\frac{S+D+I}{N}
$$

- $S,\,D,\,I$: substitutions, deletions and insertions in a minimum-edit (Levenshtein) alignment of hypothesis words to reference words
- $N$: number of reference words. Insertions are not capped, so WER can exceed 100%

> Figure: The reference is "set a timer for ten minutes please" (N = 7). The default S + D + I hypothesis has one insertion (so), one substitution (a → the) and one deletion (please), so WER = 3/7 = 42.9% and CER = 38.2%. The sound-alikes hypothesis has the same 42.9% WER but only 17.6% CER: every near-miss still costs a whole word. Under the raw normaliser the formatted hypothesis scores 57.1% (4 substitutions). Lowercasing and stripping punctuation drops it to 14.3%, and spelling out numbers drops it to 0%. The hallucinated tail adds 4 insertions, giving 57.1%.

Worked example: reference "set a timer for ten minutes please" vs hypothesis "so set the timer for ten minutes"

| Reference | Hypothesis | Operation | Running edits |
| --- | --- | --- | --- |
| · | so | insertion | 1 |
| set | set | match | 1 |
| a | the | substitution | 2 |
| timer | timer | match | 2 |
| for | for | match | 2 |
| ten | ten | match | 2 |
| minutes | minutes | match | 2 |
| please | · | deletion | 3 |

So $S=1$, $D=1$, $I=1$, $N=7$ and WER $=3/7=42.9\%$. Both strings have 7 words, so naively comparing word by word would give 6 mismatches of 7. The alignment step is essential.

**Shapes for one 30 s Whisper window**

1. $30\text{ s}\times16{,}000=480{,}000$ samples; window 400, hop 160 → $480{,}000/160=3000$ frames.
2. Log-mel input: $80\times3000=240{,}000$ numbers.
3. Two convolutions with filter width 3, the second with stride 2 → 1500 encoder positions of 20 ms each.
4. wav2vec 2.0 on the same clip: its 7-layer CNN (total stride 320) gives 1499 frames, about 50 per second.

```python
import re
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
hyp = asr("timer.wav")["text"]          # 16 kHz audio is resampled/padded to 30 s internally

def normalise(s):
    return re.sub(r"[^\w' ]", "", s.lower()).split()

def wer(ref, hyp):
    r, h = normalise(ref), normalise(hyp)
    d = list(range(len(h) + 1))                      # one row of the Levenshtein table
    for i in range(1, len(r) + 1):
        prev, d[0] = d[0], i
        for j in range(1, len(h) + 1):
            prev, d[j] = d[j], min(prev + (r[i-1] != h[j-1]), d[j] + 1, d[j-1] + 1)
    return d[-1] / len(r)

ref = "set a timer for ten minutes please"
print(wer(ref, "so set the timer for ten minutes"))     # 0.4286
print(wer(ref, "Set a timer for 10 minutes, please."))  # 0.1429 ("10" vs "ten")
```

Transcribe with Whisper (Hugging Face), then score with a small WER function

> **Normalise before you compare:** WER compares strings, so casing, punctuation, "10" vs "ten" and "Mr." vs "mister" all count as errors unless both sides are normalised the same way. Whisper's paper used a dedicated text normaliser for this reason. Two WER numbers are only comparable if they used the same normaliser and the same test set.

> **Hallucination is the attention decoder's failure mode:** A CTC or transducer model can only emit labels while consuming frames, so silence tends to produce blanks. An attention decoder is a conditional language model: on silence or noise it can produce fluent text that was never said, or repeat a phrase. Voice-activity detection, timestamp constraints and repetition checks are the usual guards.

**Strengths**

- CTC: parallel, fast and monotonic; pairs well with self-supervised encoders
- Attention encoder–decoder: best accuracy offline; one model can handle language ID, translation and timestamps
- RNN-T: streaming with label context; the standard for on-device dictation

**Limitations**

- CTC: conditionally independent frames; needs an LM for good spelling
- Attention: needs the whole window, autoregressive, can hallucinate or loop
- RNN-T: memory-heavy T × U joint lattice in training

**Question:** Reference has 5 words. The hypothesis contains all 5, correctly, plus 6 extra words. What is the WER?

- A. 0%, since every reference word was recognised
- B. 54.5%
- C. 100%
- D. 120%

**Answer:** D. 120% — $S=D=0$ and $I=6$, so WER $=6/5=120\%$. The denominator is the reference length, not the hypothesis length, so insertions can push WER above 100%.

**Question:** Why does an RNN-T stream more naturally than an attention encoder–decoder?

- A. It uses a smaller vocabulary
- B. Its decoding advances monotonically over encoder frames, emitting labels or blank at each one, so it never needs future audio
- C. It has no encoder
- D. It is trained without transcripts

**Answer:** B. Its decoding advances monotonically over encoder frames, emitting labels or blank at each one, so it never needs future audio — The transducer lattice moves either to the next frame (blank) or to the next label, so output depends only on frames seen so far, given a causal encoder. An attention decoder cross-attends to the whole utterance's encoder output.

**Key points**

- Input: 80-bin log-mel, 25 ms window, 10 ms hop → 100 frames/s; mel $m(f)=2595\log_{10}(1+f/700)$.
- Decoders: CTC (frame-wise plus collapse), attention encoder–decoder (LAS, Whisper), RNN-T (streaming).
- wav2vec 2.0: contrastive self-supervision on raw audio; 4.8/8.2 LibriSpeech WER from 10 minutes of labels.
- Whisper: 680k hours of weak supervision, 30 s windows, task chosen by special tokens.
- WER $=(S+D+I)/N$ from a Levenshtein alignment; normalise text first; it can exceed 100%.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
