Definition
Short-time Fourier transform is the sequence of DFTs of short, overlapping, windowed frames of a signal, indexed by frame (time) and bin (frequency). Its magnitude is the spectrogram, the standard input of speech and audio models. The time–frequency view goes back to Gabor, "Theory of Communication" (1946). The analysis–synthesis framework was set out by Allen and Rabiner (Proc. IEEE, 1977). Griffin and Lim, "Signal Estimation from Modified Short-Time Fourier Transform" (IEEE Trans. ASSP, 1984), showed how to rebuild a signal from magnitudes alone.
Speech and music change over time. One Fourier transform of a whole utterance gives its overall frequency content but says nothing about when each frequency occurs. A rising pitch and a falling pitch can have the same spectrum. The STFT applies the transform locally: over 20–40 ms speech is close to stationary, so a frame's spectrum describes one phone-sized slice.
Definition and parameters
n_fft: FFT size; gives bins spaced- window of
win_lengthsamples (zero-padded to in the centre), usually periodic Hann hop_length: samples between frame starts; the frame rate is- frame index; with centring, frame is centred on sample
The speech default is a 25 ms window every 10 ms. At 16 kHz this is n_fft = win_length = 400 and hop = 160, giving 201 bins 40 Hz apart and 100 frames per second; consecutive frames overlap by 240 samples (60%). With centring (center=True in torch and librosa) the signal is padded by on both sides, so the frame count depends only on the hop:
- signal length in samples; 1 s at 16 kHz with , gives 101 or 98 frames
| Setting | fs | n_fft / win | hop | Notes |
|---|---|---|---|---|
| ASR front end (Kaldi-style, Whisper) | 16 kHz | 400 / 400 (25 ms) | 160 (10 ms) | Whisper: Hann, 80 mel bands (128 in large-v3), 30 s → 3000 frames |
| Tacotron 2 | 24 kHz | 50 ms window | 12.5 ms | 80 mel bands, 125 Hz – 7.6 kHz |
| HiFi-GAN (LJSpeech) | 22.05 kHz | 1024 / 1024 (46.4 ms) | 256 (11.6 ms) | 80 mel bands, fmax 8 kHz |
| librosa defaults | any | 2048 / 2048 | 512 | hop = win_length / 4 |
| torch.stft defaults | any | n_fft required | n_fft // 4 | window = None means rectangular |
The time–frequency trade-off
Bin spacing is and the window lasts , so their product is always 1. A long window separates close partials but smears events in time. A short window localises clicks and onsets but blurs harmonics. Gabor's uncertainty relation makes this a law rather than an engineering limit. For the RMS widths of any window in time and in frequency, , with equality only for a Gaussian.
Speech settings sit in between. A 25 ms window holds 2–3 pitch periods of a 100 Hz voice and resolves formants, and a 10 ms hop follows changes between phones. Music analysis often uses longer windows (2048 samples at 44.1 kHz = 46 ms) to resolve notes.
Inversion: overlap-add
Each column can be inverted with an IFFT, which returns the windowed frame. Multiplying again by a synthesis window, adding the frames at their offsets and dividing by the summed squared window recovers exactly:
- window envelope; must never be 0 (the NOLA condition)
- stronger: constant. Periodic Hann sums to 1 at and to 2 at
The speech setting 400 / 160 is not COLA for Hann: the window sum ripples between 1.19 and 1.31. It is NOLA, so torch.istft still inverts it exactly, because it divides by the envelope.
Magnitude, power and the missing phase
Models usually take the magnitude or the power , then a mel filterbank and a log; see Mel spectrograms. In dB the two are identical, ; with a natural log, log power is twice log magnitude. Dropping phase halves the data, and the magnitude is stable under small shifts of the frame.
To produce audio from a magnitude, a phase must be invented. Griffin–Lim starts from random phase and alternates: ISTFT to a signal, STFT again, keep the new phase, and restore the target magnitude. Each iteration does not increase . Tacotron (2017) used it as its vocoder. Its metallic, phasey artefacts are why WaveNet, WaveRNN and HiFi-GAN vocoders replaced it.
Worked example: shapes at 16 kHz, n_fft 400, hop 160
| Input | Frames T | Spectrogram | Note |
|---|---|---|---|
| 1 s (16,000 samples) | 1 + ⌊16000/160⌋ = 101 | 201 × 101 complex = 20,301 | frame rate 100 per second |
| 1 s, center=False | 1 + ⌊15600/160⌋ = 98 | 201 × 98 | frames end at sample 15,920; the last 80 samples are in no frame |
| 2 s | 1 + 32000/160 = 201 | 201 × 201 | T grows by exactly 100 per second |
| 30 s Whisper window | 1 + 480000/160 = 3001 | Whisper drops the last: 80 mel × 3000 | encoder conv stride 2 → 1500 |
| Redundancy | 201 complex per 160 new samples | 402 reals / 160 = 2.51× | so not every array is a valid STFT |
1import numpy as np23def stft(x, n_fft=400, hop=160):4 """Like torch.stft(center=True, pad_mode='reflect', window=hann_window(n_fft))."""5 w = 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(n_fft) / n_fft) # periodic Hann6 xp = np.pad(x, n_fft // 2, mode="reflect")7 frames = np.lib.stride_tricks.sliding_window_view(xp, n_fft)[::hop] # (T, n_fft)8 return np.fft.rfft(frames * w, axis=1).T, w # (n_fft/2 + 1, T)910def istft(S, w, hop, length):11 """Overlap-add with window-envelope normalisation (needs NOLA, not COLA)."""12 n_fft = len(w)13 frames = np.fft.irfft(S.T, n=n_fft, axis=1) * w14 out = np.zeros(n_fft + hop * (len(frames) - 1))15 env = np.zeros_like(out)16 for m, f in enumerate(frames):17 out[m * hop:m * hop + n_fft] += f18 env[m * hop:m * hop + n_fft] += w ** 219 s = slice(n_fft // 2, n_fft // 2 + length) # drop the centre padding20 return out[s] / env[s]2122fs = 16_00023t = np.arange(fs) / fs24x = np.sin(2 * np.pi * (300 * t + 1_500 * t ** 2)) # chirp 300 -> 3300 Hz in 1 s25S, w = stft(x)26print(S.shape) # (201, 101)27power_db = 10 * np.log10(np.abs(S) ** 2 + 1e-10) # power spectrogram in dB28print(power_db[:, 50].argmax() * fs / 400, "Hz") # 1800.0 at t = 0.5 s29print(np.abs(istft(S, w, 160, len(x)) - x).max()) # ~9e-16: exact inversionCheck yourself
A 2 s clip at 16 kHz goes through torch.stft(n_fft=400, hop_length=160, center=True). What is the output shape?
Check yourself
You must tell apart two clicks 6 ms apart at 8 kHz. Which window works?
Remember
- ✓STFT: DFT of windowed frames ; output complex.
- ✓Speech default at 16 kHz: n_fft = win = 400 (25 ms), hop 160 (10 ms), Hann → 201 bins at 40 Hz, 100 frames/s.
- ✓Frames: 1 + ⌊L/hop⌋ with centring (N/2 padding), 1 + ⌊(L − n_fft)/hop⌋ without; Whisper's 30 s gives 3001, trimmed to 3000.
- ✓Window length × bin spacing = 1: long windows resolve frequency, short windows resolve time (Gabor: σ_t·σ_f ≥ 1/4π).
- ✓Overlap-add with division by Σw² inverts exactly whenever the envelope is non-zero (NOLA); Hann at hop N/2 or N/4 is also COLA.
- ✓Magnitude or power spectrograms drop phase; Griffin–Lim (1984) or a neural vocoder must supply it to make audio.