# Whisper

> An encoder–decoder Transformer trained on 680,000 h of weakly labelled web audio. It reads 30 s log-mel windows and writes text; special tokens choose the language, the task and timestamps.

Source: https://www.mlfoundry.in/learn/whisper  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Whisper:** is OpenAI's speech recognition and translation model, from Radford, Kim, Xu, Brockman, McLeavey and Sutskever, "Robust Speech Recognition via Large-Scale Weak Supervision" (2022; ICML 2023). It is a standard encoder–decoder Transformer. It is trained with plain next-token prediction on **680,000 hours** of audio paired with transcripts found on the web. A single model does multilingual transcription, translation of any language into English, language identification, voice-activity detection and timestamping. A few special tokens in the decoder prompt choose which of these it does.

The problem is robustness. Models trained on one curated corpus, such as the 960 h of LibriSpeech audiobooks, reach low error on that test set but degrade badly on phone calls, accents, noise or other domains. Self-supervised encoders such as **wav2vec 2.0** learn good audio features without labels, but they still need a fine-tuning stage, and the fine-tuned model inherits the quirks of its fine-tuning set.

Whisper's bet was the opposite: collect a very large, noisy, *labelled* dataset and use no fine-tuning at all. Transcripts that looked machine-generated (for example all upper case, or with no punctuation) were filtered out heuristically. A language detector also checked that the spoken language matched the transcript's language. The 680,000 h split into 438,000 h of English transcription, 125,000 h of X→English translation and 117,000 h of transcription in 96 other languages. The whole system is evaluated **zero-shot** on benchmarks it never trained on.

## Architecture

Audio is resampled to 16 kHz and cut or zero-padded to exactly **30 s**. The front end computes an 80-channel log-magnitude mel spectrogram with 25 ms windows and a 10 ms hop (see **Mel spectrograms**), giving $80\times3000$ values, scaled to roughly $[-1,1]$. Large-v3 (2023) uses 128 mel channels instead. A stem of two 1-D convolutions (filter width 3, GELU) follows. The second convolution has stride 2, so 3000 frames become 1500 encoder positions, one per 20 ms.

$$
T_{\text{enc}}=\Big\lfloor\frac{3000+2\cdot1-3}{2}\Big\rfloor+1=1500,\qquad p(y\mid x)=\prod_{u}p\big(y_u\mid y_{<u},\,\text{prompt},\,\mathrm{Enc}(x)\big)
$$

- $T_{\text{enc}}$: encoder length after the stride-2 convolution (kernel 3, padding 1); each position covers 20 ms
- $\text{prompt}$: special tokens (and optionally previous text) that set the task before any text is generated
- $\mathrm{Enc}(x)$: the 1500 encoder outputs; every decoder layer reads them through **Cross-attention**

> Figure: Whisper end to end. A 30 s, 16 kHz window (480,000 samples) becomes an 80 × 3000 log-mel spectrogram. A conv stem with stride 2 halves it to 1500 frames, a Transformer encoder processes them, and the decoder cross-attends to all 1500 while predicting tokens. The decoder sequence starts with the task prompt (start, language, transcribe or translate). Text segments are then bracketed by timestamp tokens in 0.02 s steps. Encoder and decoder have the same depth and width; sizes range from tiny (39M) to large (1550M).

Encoder and decoder have the same width and number of blocks. Both use pre-activation (pre-LN) residual blocks. The encoder adds sinusoidal position embeddings, and the decoder learns its positions and ties its input and output token embeddings. Text uses GPT-2's byte-level BPE (see **Byte-pair encoding (BPE)**) for the English-only models. The multilingual models use a vocabulary of the same size refit on multilingual data. The decoder context is 448 tokens.

## One model, many tasks: the token format

Every training example is written in one format. It starts with `<|startoftranscript|>`, then a language tag such as `<|en|>` (99 languages in the tokenizer). The decoder can predict this tag itself, which is how Whisper does language identification. A no-speech token marks windows without speech. Next comes `<|transcribe|>` or `<|translate|>` (into English), and then either text with timestamps or `<|notimestamps|>` followed by text only. The sequence ends with `<|endoftext|>`.

Timestamps are ordinary vocabulary tokens: `<|0.00|>`, `<|0.02|>`, … `<|30.00|>`, measured from the start of the window. The 0.02 s step is exactly one encoder frame ($30/1500$), so there are $30/0.02+1=1501$ of them. Each segment is written as start time, text, end time. Optionally, `<|startofprev|>` and the previous window's transcript come before the prompt, so the decoder can keep names and style consistent from one window to the next.

Worked example: the window arithmetic for the second window of the interactive figure below (window starts at 26.1 s)

| Quantity | Computation | Value |
| --- | --- | --- |
| samples per window | 30 s × 16,000 | 480,000 |
| mel frames | 480,000 / 160 hop | 3000 |
| encoder positions | ⌊(3000 + 2 − 3)/2⌋ + 1 | 1500 (20 ms each) |
| timestamp tokens | 30 / 0.02 + 1 | 1501 |
| segment 26.5–30.8 s, relative | 26.5 − 26.1, 30.8 − 26.1 | <\|0.40\|> … <\|4.70\|> |
| their token indices | 0.40 / 0.02, 4.70 / 0.02 | #20, #235 |
| last closed timestamp | <\|24.50\|> = #1225 | next window starts at 26.1 + 24.5 = 50.6 s |

**Sizes and results**

- **Sizes:** tiny 39M (4 layers, width 384, 6 heads) · base 74M (6, 512, 8) · small 244M (12, 768, 12) · medium 769M (24, 1024, 16) · large 1550M (32, 1280, 20)
- **English-only:** tiny.en to medium.en: the same sizes trained on English only, slightly better for English at the small sizes
- **Robustness:** zero-shot large-v2 matches a supervised wav2vec 2.0 Large at 2.7% WER on LibriSpeech test-clean, yet makes on average 55.2% fewer errors across the other English test sets in the paper
- **Versions:** large-v2 (Dec 2022): same architecture, trained longer; large-v3 (Nov 2023): 128 mel bins; large-v3-turbo (2024): decoder cut from 32 to 4 layers, 809M
- **Front end:** 16 kHz, 25 ms / 10 ms, 80 log-mel bins (128 in large-v3), 30 s windows, 1500 encoder frames, 448 decoder tokens

## Long audio: windows that move by timestamps

The model only ever sees 30 s at a time, so long recordings are transcribed in a loop. After decoding a window, Whisper looks at the last timestamp. If the final segment was closed, the next window starts a full 30 s later. If a segment was still open when the window ended, the incomplete segment is thrown away, and the next window starts at the last **closed** timestamp. That sentence is then decoded again from its beginning. Without timestamps you can only cut fixed chunks, and any sentence that crosses a chunk edge is heard in two halves.

> Figure: A scripted 98 s talk of 21 sentences, cut into Whisper windows. Default (98 s, timestamps + seek): 4 windows. Window 1 ends inside "the decoder writes text…", so window 2 starts at its last closed timestamp, 26.1 s. Window 2 ends inside a sentence too and hands over at <|24.50|> (token #1225), i.e. 50.6 s. 9.4 s of audio is decoded twice, no sentence is split, and the last window is padded with 12.6 s of silence (1260 mel frames). Switch to fixed 30 s chunks: still 4 windows, but 3 sentences are split at 30, 60 and 90 s, and the last chunk is padded with 22.0 s. At 30 s of audio a single window suffices and nothing is cut. "previous text" adds the <|startofprev|> prompt.

The paper adds a set of decoding heuristics for long-form audio. It uses beam search with 5 beams. If the output's gzip compression ratio exceeds 2.4 (a sign of repetition) or its average log-probability falls below −1, the window is decoded again at a higher temperature, stepping 0, 0.2, … 1.0. A window is treated as silence when the no-speech probability exceeds 0.6 and the average log-probability is also below −1. The first timestamp is constrained to fall within the first second. The previous window's text is used as the prompt.

```python
import whisper

model = whisper.load_model("small")                     # 244M, multilingual
audio = whisper.load_audio("talk.wav")                  # float32 mono, resampled to 16 kHz
window = whisper.pad_or_trim(audio)                     # exactly 480,000 samples
mel = whisper.log_mel_spectrogram(window, n_mels=model.dims.n_mels).to(model.device)
print(mel.shape)                                        # torch.Size([80, 3000]); 128 rows for large-v3

_, probs = model.detect_language(mel)                   # the decoder predicts the <|lang|> token
print(max(probs, key=probs.get))

result = model.transcribe(
    "talk.wav", task="transcribe", beam_size=5,
    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0),         # fallback ladder
    compression_ratio_threshold=2.4, logprob_threshold=-1.0,
    no_speech_threshold=0.6, condition_on_previous_text=True)
for s in result["segments"]:
    print(f'{s["start"]:7.2f} {s["end"]:7.2f} {s["text"]}')
```

The openai-whisper package: the 30 s front end, language detection, and long-form transcription with the paper's fallback thresholds (not run here: needs torch and ffmpeg)

> **Hallucination and runaway repetition:** The decoder is a conditional language model. On silence, music or noise it can write fluent text that was never spoken, such as "Thanks for watching!", a pattern common in web subtitles. It can also loop the same phrase. Conditioning on previous text makes this worse once an error appears, because the mistake is fed back as context. Practical guards: run a voice-activity detector first, keep the compression-ratio and no-speech checks, and set `condition_on_previous_text=False` if errors propagate.

> **Why timestamps are tokens:** Because times are tokens, the same next-token loss that learns spelling also learns alignment, and timestamps cost no extra head. Their 20 ms grid matches the encoder frame, so a timestamp effectively points at an encoder position. Segment-level timestamps from the decoder are often off by a fraction of a second. Word-level times are usually recovered afterwards, by aligning cross-attention weights or with a separate forced aligner.

> **Smaller and faster descendants:** Most of Whisper's inference time is spent in the autoregressive decoder, not the encoder. Distil-Whisper (Gandhi, von Platen and Rush, 2023) keeps the full encoder, keeps only two decoder layers, and trains them to imitate large-v2 on pseudo-labels (see **Knowledge distillation**). large-v3-turbo takes the same route with four decoder layers. Both trade a little accuracy for several times faster decoding.

**Strengths**

- Zero-shot robustness across accents, noise and domains, with no fine-tuning needed
- One model for 99-language transcription, translation into English, language ID and timestamps
- Open weights at five sizes; the 30 s format is simple to batch

**Limitations**

- Not streaming: it needs a whole 30 s window, and the decoder is autoregressive
- Hallucinates on silence or noise and can loop; long-form needs heuristics
- Accuracy drops sharply for languages with little training data

**Question:** A window ends while a sentence is still being spoken. What does Whisper's long-form loop do?

- A. Keeps the half sentence and starts the next window exactly 30 s later
- B. Drops the unfinished segment and starts the next window at the last closed timestamp, so the sentence is decoded again in full
- C. Extends the window beyond 30 s until the sentence ends
- D. Switches to <|notimestamps|> for that window

**Answer:** B. Drops the unfinished segment and starts the next window at the last closed timestamp, so the sentence is decoded again in full — The encoder input is always exactly 30 s, so the window cannot grow. Instead the next window starts (seeks) to the last closed timestamp. The cost is re-decoding the overlap; the gain is that no sentence is split.

**Question:** Why are there exactly 1501 timestamp tokens?

- A. The vocabulary had 1501 unused slots
- B. Times from 0.00 to 30.00 s in steps of 0.02 s, the length of one encoder frame (30 / 1500), give 30 / 0.02 + 1 values
- C. One per mel frame
- D. One per millisecond of the first 1.5 s

**Answer:** B. Times from 0.00 to 30.00 s in steps of 0.02 s, the length of one encoder frame (30 / 1500), give 30 / 0.02 + 1 values — The stride-2 stem turns 3000 mel frames into 1500 encoder positions of 20 ms. Timestamps are quantised to that grid, from `<|0.00|>` to `<|30.00|>` inclusive: $1500+1=1501$.

**Key points**

- Weak supervision at scale: 680,000 h of web audio with transcripts, no fine-tuning, evaluated zero-shot.
- 30 s windows → 80 × 3000 log-mel → stride-2 conv stem → 1500 encoder frames of 20 ms; the decoder cross-attends to all of them.
- Task by prompt: `<|startoftranscript|>`, language, `<|transcribe|>`/`<|translate|>`, timestamps or `<|notimestamps|>`.
- Timestamps are 1501 tokens on a 0.02 s grid; long audio advances to the last closed timestamp, so no sentence is split.
- Sizes from tiny 39M to large 1550M; failure modes are hallucination and repetition on silence or noise.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
