Definition
Whisper is OpenAI's speech recognition and translation model, from Radford, Kim, Xu, Brockman, McLeavey and Sutskever, "Robust Speech Recognition via Large-Scale Weak Supervision" (2022; ICML 2023). It is a standard encoder–decoder Transformer. It is trained with plain next-token prediction on 680,000 hours of audio paired with transcripts found on the web. A single model does multilingual transcription, translation of any language into English, language identification, voice-activity detection and timestamping. A few special tokens in the decoder prompt choose which of these it does.
The problem is robustness. Models trained on one curated corpus, such as the 960 h of LibriSpeech audiobooks, reach low error on that test set but degrade badly on phone calls, accents, noise or other domains. Self-supervised encoders such as wav2vec 2.0 learn good audio features without labels, but they still need a fine-tuning stage, and the fine-tuned model inherits the quirks of its fine-tuning set.
Whisper's bet was the opposite: collect a very large, noisy, labelled dataset and use no fine-tuning at all. Transcripts that looked machine-generated (for example all upper case, or with no punctuation) were filtered out heuristically. A language detector also checked that the spoken language matched the transcript's language. The 680,000 h split into 438,000 h of English transcription, 125,000 h of X→English translation and 117,000 h of transcription in 96 other languages. The whole system is evaluated zero-shot on benchmarks it never trained on.
Architecture
Audio is resampled to 16 kHz and cut or zero-padded to exactly 30 s. The front end computes an 80-channel log-magnitude mel spectrogram with 25 ms windows and a 10 ms hop (see Mel spectrograms), giving values, scaled to roughly . Large-v3 (2023) uses 128 mel channels instead. A stem of two 1-D convolutions (filter width 3, GELU) follows. The second convolution has stride 2, so 3000 frames become 1500 encoder positions, one per 20 ms.
- encoder length after the stride-2 convolution (kernel 3, padding 1); each position covers 20 ms
- special tokens (and optionally previous text) that set the task before any text is generated
- the 1500 encoder outputs; every decoder layer reads them through Cross-attention
Encoder and decoder have the same width and number of blocks. Both use pre-activation (pre-LN) residual blocks. The encoder adds sinusoidal position embeddings, and the decoder learns its positions and ties its input and output token embeddings. Text uses GPT-2's byte-level BPE (see Byte-pair encoding (BPE)) for the English-only models. The multilingual models use a vocabulary of the same size refit on multilingual data. The decoder context is 448 tokens.
One model, many tasks: the token format
Every training example is written in one format. It starts with <|startoftranscript|>, then a language tag such as <|en|> (99 languages in the tokenizer). The decoder can predict this tag itself, which is how Whisper does language identification. A no-speech token marks windows without speech. Next comes <|transcribe|> or <|translate|> (into English), and then either text with timestamps or <|notimestamps|> followed by text only. The sequence ends with <|endoftext|>.
Timestamps are ordinary vocabulary tokens: <|0.00|>, <|0.02|>, … <|30.00|>, measured from the start of the window. The 0.02 s step is exactly one encoder frame (), so there are of them. Each segment is written as start time, text, end time. Optionally, <|startofprev|> and the previous window's transcript come before the prompt, so the decoder can keep names and style consistent from one window to the next.
| Quantity | Computation | Value |
|---|---|---|
| samples per window | 30 s × 16,000 | 480,000 |
| mel frames | 480,000 / 160 hop | 3000 |
| encoder positions | ⌊(3000 + 2 − 3)/2⌋ + 1 | 1500 (20 ms each) |
| timestamp tokens | 30 / 0.02 + 1 | 1501 |
| segment 26.5–30.8 s, relative | 26.5 − 26.1, 30.8 − 26.1 | <|0.40|> … <|4.70|> |
| their token indices | 0.40 / 0.02, 4.70 / 0.02 | #20, #235 |
| last closed timestamp | <|24.50|> = #1225 | next window starts at 26.1 + 24.5 = 50.6 s |
Sizes and results
- Sizes
- tiny 39M (4 layers, width 384, 6 heads) · base 74M (6, 512, 8) · small 244M (12, 768, 12) · medium 769M (24, 1024, 16) · large 1550M (32, 1280, 20)
- English-only
- tiny.en to medium.en: the same sizes trained on English only, slightly better for English at the small sizes
- Robustness
- zero-shot large-v2 matches a supervised wav2vec 2.0 Large at 2.7% WER on LibriSpeech test-clean, yet makes on average 55.2% fewer errors across the other English test sets in the paper
- Versions
- large-v2 (Dec 2022): same architecture, trained longer; large-v3 (Nov 2023): 128 mel bins; large-v3-turbo (2024): decoder cut from 32 to 4 layers, 809M
- Front end
- 16 kHz, 25 ms / 10 ms, 80 log-mel bins (128 in large-v3), 30 s windows, 1500 encoder frames, 448 decoder tokens
Long audio: windows that move by timestamps
The model only ever sees 30 s at a time, so long recordings are transcribed in a loop. After decoding a window, Whisper looks at the last timestamp. If the final segment was closed, the next window starts a full 30 s later. If a segment was still open when the window ended, the incomplete segment is thrown away, and the next window starts at the last closed timestamp. That sentence is then decoded again from its beginning. Without timestamps you can only cut fixed chunks, and any sentence that crosses a chunk edge is heard in two halves.
The paper adds a set of decoding heuristics for long-form audio. It uses beam search with 5 beams. If the output's gzip compression ratio exceeds 2.4 (a sign of repetition) or its average log-probability falls below −1, the window is decoded again at a higher temperature, stepping 0, 0.2, … 1.0. A window is treated as silence when the no-speech probability exceeds 0.6 and the average log-probability is also below −1. The first timestamp is constrained to fall within the first second. The previous window's text is used as the prompt.
1import whisper23model = whisper.load_model("small") # 244M, multilingual4audio = whisper.load_audio("talk.wav") # float32 mono, resampled to 16 kHz5window = whisper.pad_or_trim(audio) # exactly 480,000 samples6mel = whisper.log_mel_spectrogram(window, n_mels=model.dims.n_mels).to(model.device)7print(mel.shape) # torch.Size([80, 3000]); 128 rows for large-v389_, probs = model.detect_language(mel) # the decoder predicts the <|lang|> token10print(max(probs, key=probs.get))1112result = model.transcribe(13 "talk.wav", task="transcribe", beam_size=5,14 temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), # fallback ladder15 compression_ratio_threshold=2.4, logprob_threshold=-1.0,16 no_speech_threshold=0.6, condition_on_previous_text=True)17for s in result["segments"]:18 print(f'{s["start"]:7.2f} {s["end"]:7.2f} {s["text"]}')Use when
- +Zero-shot robustness across accents, noise and domains, with no fine-tuning needed
- +One model for 99-language transcription, translation into English, language ID and timestamps
- +Open weights at five sizes; the 30 s format is simple to batch
Avoid when
- −Not streaming: it needs a whole 30 s window, and the decoder is autoregressive
- −Hallucinates on silence or noise and can loop; long-form needs heuristics
- −Accuracy drops sharply for languages with little training data
Check yourself
A window ends while a sentence is still being spoken. What does Whisper's long-form loop do?
Check yourself
Why are there exactly 1501 timestamp tokens?
Remember
- ✓Weak supervision at scale: 680,000 h of web audio with transcripts, no fine-tuning, evaluated zero-shot.
- ✓30 s windows → 80 × 3000 log-mel → stride-2 conv stem → 1500 encoder frames of 20 ms; the decoder cross-attends to all of them.
- ✓Task by prompt:
<|startoftranscript|>, language,<|transcribe|>/<|translate|>, timestamps or<|notimestamps|>. - ✓Timestamps are 1501 tokens on a 0.02 s grid; long audio advances to the last closed timestamp, so no sentence is split.
- ✓Sizes from tiny 39M to large 1550M; failure modes are hallucination and repetition on silence or noise.