Whisper

An encoder–decoder Transformer trained on 680,000 h of weakly labelled web audio. It reads 30 s log-mel windows and writes text; special tokens choose the language, the task and timestamps.

8 min read999 wordscore0 viewsupdated 2026-10-15

Definition

Whisper is OpenAI's speech recognition and translation model, from Radford, Kim, Xu, Brockman, McLeavey and Sutskever, "Robust Speech Recognition via Large-Scale Weak Supervision" (2022; ICML 2023). It is a standard encoder–decoder Transformer. It is trained with plain next-token prediction on 680,000 hours of audio paired with transcripts found on the web. A single model does multilingual transcription, translation of any language into English, language identification, voice-activity detection and timestamping. A few special tokens in the decoder prompt choose which of these it does.

The problem is robustness. Models trained on one curated corpus, such as the 960 h of LibriSpeech audiobooks, reach low error on that test set but degrade badly on phone calls, accents, noise or other domains. Self-supervised encoders such as wav2vec 2.0 learn good audio features without labels, but they still need a fine-tuning stage, and the fine-tuned model inherits the quirks of its fine-tuning set.

Whisper's bet was the opposite: collect a very large, noisy, labelled dataset and use no fine-tuning at all. Transcripts that looked machine-generated (for example all upper case, or with no punctuation) were filtered out heuristically. A language detector also checked that the spoken language matched the transcript's language. The 680,000 h split into 438,000 h of English transcription, 125,000 h of X→English translation and 117,000 h of transcription in 96 other languages. The whole system is evaluated zero-shot on benchmarks it never trained on.

Architecture

Audio is resampled to 16 kHz and cut or zero-padded to exactly 30 s. The front end computes an 80-channel log-magnitude mel spectrogram with 25 ms windows and a 10 ms hop (see Mel spectrograms), giving 80×300080\times3000 values, scaled to roughly [−1,1][-1,1]. Large-v3 (2023) uses 128 mel channels instead. A stem of two 1-D convolutions (filter width 3, GELU) follows. The second convolution has stride 2, so 3000 frames become 1500 encoder positions, one per 20 ms.

Tenc=⌊3000+2⋅1−32⌋+1=1500,p(y∣x)=∏up(yu∣y<u, prompt, Enc(x))T_{\text{enc}}=\Big\lfloor\frac{3000+2\cdot1-3}{2}\Big\rfloor+1=1500,\qquad p(y\mid x)=\prod_{u}p\big(y_u\mid y_{<u},\,\text{prompt},\,\mathrm{Enc}(x)\big)
TencT_{\text{enc}}
encoder length after the stride-2 convolution (kernel 3, padding 1); each position covers 20 ms
prompt\text{prompt}
special tokens (and optionally previous text) that set the task before any text is generated
Enc(x)\mathrm{Enc}(x)
the 1500 encoder outputs; every decoder layer reads them through Cross-attention
30 s @ 16 kHz480,000 sampleslog-mel 80 × 3000conv stem2 × conv1d, width 3GELU; 2nd stride 23000 → 1500 frames+ sinusoidal pos.encoder × Npre-LN Transformerself-attention overall 1500 frames(20 ms each)model sizes (encoder = decoder depth)layerswidthheadsparamstiny4384639Mbase6512874Msmall1276812244Mmedium24102416769Mlarge321280201550Mkeys, valuesdecoder × Nmasked self-attention over tokens so farcross-attention to the 1500 encoder frameslearned positions · 448-token contextnext-token softmax (tied embeddings)decoder sequence: task prompt, then timestamped segments (times relative to the window)<|startoftranscript|><|en|><|transcribe|><|0.60|>welcome back to the course<|4.00|><|4.50|>today we look at speech recognition<|8.70|>…<|endoftext|>language: one of 99 tags · task: <|transcribe|> or <|translate|> (into English)timestamps: <|0.00|> … <|30.00|> in 0.02 s steps (1501 tokens), or <|notimestamps|> for text only680,000 h of weakly labelled web audio, one model for all tasks438,000 h English transcription · 125,000 h X→English translation · 117,000 h in 96 other languages
fig.Whisper end to end. A 30 s, 16 kHz window (480,000 samples) becomes an 80 × 3000 log-mel spectrogram. A conv stem with stride 2 halves it to 1500 frames, a Transformer encoder processes them, and the decoder cross-attends to all 1500 while predicting tokens. The decoder sequence starts with the task prompt (start, language, transcribe or translate). Text segments are then bracketed by timestamp tokens in 0.02 s steps. Encoder and decoder have the same depth and width; sizes range from tiny (39M) to large (1550M).

Encoder and decoder have the same width and number of blocks. Both use pre-activation (pre-LN) residual blocks. The encoder adds sinusoidal position embeddings, and the decoder learns its positions and ties its input and output token embeddings. Text uses GPT-2's byte-level BPE (see Byte-pair encoding (BPE)) for the English-only models. The multilingual models use a vocabulary of the same size refit on multilingual data. The decoder context is 448 tokens.

One model, many tasks: the token format

Every training example is written in one format. It starts with <|startoftranscript|>, then a language tag such as <|en|> (99 languages in the tokenizer). The decoder can predict this tag itself, which is how Whisper does language identification. A no-speech token marks windows without speech. Next comes <|transcribe|> or <|translate|> (into English), and then either text with timestamps or <|notimestamps|> followed by text only. The sequence ends with <|endoftext|>.

Timestamps are ordinary vocabulary tokens: <|0.00|>, <|0.02|>, … <|30.00|>, measured from the start of the window. The 0.02 s step is exactly one encoder frame (30/150030/1500), so there are 30/0.02+1=150130/0.02+1=1501 of them. Each segment is written as start time, text, end time. Optionally, <|startofprev|> and the previous window's transcript come before the prompt, so the decoder can keep names and style consistent from one window to the next.

QuantityComputationValue
samples per window30 s × 16,000480,000
mel frames480,000 / 160 hop3000
encoder positions⌊(3000 + 2 − 3)/2⌋ + 11500 (20 ms each)
timestamp tokens30 / 0.02 + 11501
segment 26.5–30.8 s, relative26.5 − 26.1, 30.8 − 26.1<|0.40|> … <|4.70|>
their token indices0.40 / 0.02, 4.70 / 0.02#20, #235
last closed timestamp<|24.50|> = #1225next window starts at 26.1 + 24.5 = 50.6 s
Worked example: the window arithmetic for the second window of the interactive figure below (window starts at 26.1 s)

Sizes and results

Sizes
tiny 39M (4 layers, width 384, 6 heads) · base 74M (6, 512, 8) · small 244M (12, 768, 12) · medium 769M (24, 1024, 16) · large 1550M (32, 1280, 20)
English-only
tiny.en to medium.en: the same sizes trained on English only, slightly better for English at the small sizes
Robustness
zero-shot large-v2 matches a supervised wav2vec 2.0 Large at 2.7% WER on LibriSpeech test-clean, yet makes on average 55.2% fewer errors across the other English test sets in the paper
Versions
large-v2 (Dec 2022): same architecture, trained longer; large-v3 (Nov 2023): 128 mel bins; large-v3-turbo (2024): decoder cut from 32 to 4 layers, 809M
Front end
16 kHz, 25 ms / 10 ms, 80 log-mel bins (128 in large-v3), 30 s windows, 1500 encoder frames, 448 decoder tokens

Long audio: windows that move by timestamps

The model only ever sees 30 s at a time, so long recordings are transcribed in a loop. After decoding a window, Whisper looks at the last timestamp. If the final segment was closed, the next window starts a full 30 s later. If a segment was still open when the window ended, the incomplete segment is thrown away, and the next window starts at the last closed timestamp. That sentence is then decoded again from its beginning. Without timestamps you can only cut fixed chunks, and any sentence that crosses a chunk edge is heard in two halves.

98 s of speech · windows moved by predicted timestamps0102030405060708090w1w2w3w44 windows · re-decoded overlap 9.4 s (orange = next seek) · last window padded 12.6 s (1260 frames)w2: audio 26.1–56.1 s → 80 × 3000 log-mel → 1500 encoder frames · decoder tokens:<|startoftranscript|><|en|><|transcribe|><|0.40|>the decoder writes text token by token<|4.70|><|5.90|>special tokens choose the task<|9.00|><|9.50|>timestamps are tokens too<|13.00|><|14.30|>each one is a multiple of twenty milliseconds<|19.10|><|19.80|>long audio is cut into windows<|24.50|><|24.90|>the next window starts at the last<|endoftext|>unfinished segment from 24.90: next window starts at last closed timestamp 24.50 (token #1225 of 1501)■ special ■ timestamp (0.02 s steps) ■ text (many BPE tokens per chip)
mode
window
fig.A scripted 98 s talk of 21 sentences, cut into Whisper windows. Default (98 s, timestamps + seek): 4 windows. Window 1 ends inside "the decoder writes text…", so window 2 starts at its last closed timestamp, 26.1 s. Window 2 ends inside a sentence too and hands over at <|24.50|> (token #1225), i.e. 50.6 s. 9.4 s of audio is decoded twice, no sentence is split, and the last window is padded with 12.6 s of silence (1260 mel frames). Switch to fixed 30 s chunks: still 4 windows, but 3 sentences are split at 30, 60 and 90 s, and the last chunk is padded with 22.0 s. At 30 s of audio a single window suffices and nothing is cut. "previous text" adds the <|startofprev|> prompt.

The paper adds a set of decoding heuristics for long-form audio. It uses beam search with 5 beams. If the output's gzip compression ratio exceeds 2.4 (a sign of repetition) or its average log-probability falls below −1, the window is decoded again at a higher temperature, stepping 0, 0.2, … 1.0. A window is treated as silence when the no-speech probability exceeds 0.6 and the average log-probability is also below −1. The first timestamp is constrained to fall within the first second. The previous window's text is used as the prompt.

python
1import whisper
2
3model = whisper.load_model("small") # 244M, multilingual
4audio = whisper.load_audio("talk.wav") # float32 mono, resampled to 16 kHz
5window = whisper.pad_or_trim(audio) # exactly 480,000 samples
6mel = whisper.log_mel_spectrogram(window, n_mels=model.dims.n_mels).to(model.device)
7print(mel.shape) # torch.Size([80, 3000]); 128 rows for large-v3
8
9_, probs = model.detect_language(mel) # the decoder predicts the <|lang|> token
10print(max(probs, key=probs.get))
11
12result = model.transcribe(
13 "talk.wav", task="transcribe", beam_size=5,
14 temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), # fallback ladder
15 compression_ratio_threshold=2.4, logprob_threshold=-1.0,
16 no_speech_threshold=0.6, condition_on_previous_text=True)
17for s in result["segments"]:
18 print(f'{s["start"]:7.2f} {s["end"]:7.2f} {s["text"]}')
The openai-whisper package: the 30 s front end, language detection, and long-form transcription with the paper's fallback thresholds (not run here: needs torch and ffmpeg)

Use when

  • +Zero-shot robustness across accents, noise and domains, with no fine-tuning needed
  • +One model for 99-language transcription, translation into English, language ID and timestamps
  • +Open weights at five sizes; the 30 s format is simple to batch

Avoid when

  • −Not streaming: it needs a whole 30 s window, and the decoder is autoregressive
  • −Hallucinates on silence or noise and can loop; long-form needs heuristics
  • −Accuracy drops sharply for languages with little training data

Check yourself

A window ends while a sentence is still being spoken. What does Whisper's long-form loop do?

Check yourself

Why are there exactly 1501 timestamp tokens?

Remember

  • ✓Weak supervision at scale: 680,000 h of web audio with transcripts, no fine-tuning, evaluated zero-shot.
  • ✓30 s windows → 80 × 3000 log-mel → stride-2 conv stem → 1500 encoder frames of 20 ms; the decoder cross-attends to all of them.
  • ✓Task by prompt: <|startoftranscript|>, language, <|transcribe|>/<|translate|>, timestamps or <|notimestamps|>.
  • ✓Timestamps are 1501 tokens on a 0.02 s grid; long audio advances to the last closed timestamp, so no sentence is split.
  • ✓Sizes from tiny 39M to large 1550M; failure modes are hallucination and repetition on silence or noise.
← back to Deep Learning roadmap