Short-time Fourier transform (STFT)

The STFT applies a DFT to overlapping windowed frames (n_fft 400, hop 160 at 16 kHz); window length trades time for frequency resolution, overlap-add inverts it exactly, and Griffin–Lim recovers a phase from magnitude.

8 min read739 wordscore0 viewsupdated 2026-10-15

Definition

Short-time Fourier transform is the sequence of DFTs of short, overlapping, windowed frames of a signal, indexed by frame (time) and bin (frequency). Its magnitude is the spectrogram, the standard input of speech and audio models. The time–frequency view goes back to Gabor, "Theory of Communication" (1946). The analysis–synthesis framework was set out by Allen and Rabiner (Proc. IEEE, 1977). Griffin and Lim, "Signal Estimation from Modified Short-Time Fourier Transform" (IEEE Trans. ASSP, 1984), showed how to rebuild a signal from magnitudes alone.

Speech and music change over time. One Fourier transform of a whole utterance gives its overall frequency content but says nothing about when each frequency occurs. A rising pitch and a falling pitch can have the same spectrum. The STFT applies the transform locally: over 20–40 ms speech is close to stationary, so a frame's spectrum describes one phone-sized slice.

Definition and parameters

S[k,m]=∑n=0N−1x[n+mH] w[n] e−j2πkn/N,k=0,…,N2,m=0,…,T−1S[k,m]=\sum_{n=0}^{N-1}x[n+mH]\,w[n]\,e^{-j2\pi kn/N},\qquad k=0,\dots,\tfrac{N}{2},\quad m=0,\dots,T-1
NN
n_fft: FFT size; gives N/2+1N/2+1 bins spaced fs/Nf_s/N
ww
window of win_length ≤N\le N samples (zero-padded to NN in the centre), usually periodic Hann
HH
hop_length: samples between frame starts; the frame rate is fs/Hf_s/H
mm
frame index; with centring, frame mm is centred on sample mHmH

The speech default is a 25 ms window every 10 ms. At 16 kHz this is n_fft = win_length = 400 and hop = 160, giving 201 bins 40 Hz apart and 100 frames per second; consecutive frames overlap by 240 samples (60%). With centring (center=True in torch and librosa) the signal is padded by N/2N/2 on both sides, so the frame count depends only on the hop:

Tcentre=1+⌊LH⌋,Tno centre=1+⌊L−NH⌋T_{\text{centre}}=1+\Big\lfloor\frac{L}{H}\Big\rfloor,\qquad T_{\text{no centre}}=1+\Big\lfloor\frac{L-N}{H}\Big\rfloor
LL
signal length in samples; 1 s at 16 kHz with N=400N=400, H=160H=160 gives 101 or 98 frames
1 · framing at 16 kHz: n_fft = win_length = 400 (25 ms), hop = 160 (10 ms), centre = Truepad 2000123456frame 3: samples 280…679 (400)hop 160frame m is centred on sample m·160 · overlap 240 = 60%2 · one column per frameframe m400 samples× Hann w[n](periodic)FFTn_fft = 400400 complex→ keep 201S[m, k]k = 0…200Δf = 40 Hz|S| or |S|²201 × T1 s: T =1 + 16000/160= 101without centring: T = 1 + ⌊(L − n_fft)/hop⌋ = 1 + ⌊15600/160⌋ = 98Whisper, 30 s: 1 + 480000/160 = 3001 frames, the last is dropped → 30003 · back to a waveforminverse STFT (complex S known)IFFT each column → 400 samples× synthesis window w[n]overlap-add at hop 160÷ Σₘ w²[n − m·hop] (needs NOLA:the sum is never 0)Hann COLA: hop N/2 or N/4 sumsto a constant; 400/160 gives 1.19–1.31Griffin–Lim 1984 (only |S| known)φ ← random phaserepeat: x ← ISTFT(|S| · e^{jφ}) φ ← angle(STFT(x))keeps |S|, finds a consistent phaseΣ(|STFT(x)| − |S|)² never increasesartefacts: metallic, phasey
fig.The STFT at Whisper-style settings. Top: the 16 kHz signal is padded by n_fft/2 = 200 samples and cut into 400-sample Hann windows every 160 samples; frame 3 covers samples 280–679 and is centred on sample 480. Middle: each frame becomes one column of 201 complex bins, so 1 s gives a 201 × 101 matrix (98 frames without centring). Bottom: overlap-add with window normalisation inverts a complex STFT exactly; Griffin–Lim estimates a phase when only |S| is known.
Settingfsn_fft / winhopNotes
ASR front end (Kaldi-style, Whisper)16 kHz400 / 400 (25 ms)160 (10 ms)Whisper: Hann, 80 mel bands (128 in large-v3), 30 s → 3000 frames
Tacotron 224 kHz50 ms window12.5 ms80 mel bands, 125 Hz – 7.6 kHz
HiFi-GAN (LJSpeech)22.05 kHz1024 / 1024 (46.4 ms)256 (11.6 ms)80 mel bands, fmax 8 kHz
librosa defaultsany2048 / 2048512hop = win_length / 4
torch.stft defaultsanyn_fft requiredn_fft // 4window = None means rectangular
STFT settings used by well-known models and libraries

The time–frequency trade-off

Bin spacing is fs/Nf_s/N and the window lasts N/fsN/f_s, so their product is always 1. A long window separates close partials but smears events in time. A short window localises clicks and onsets but blurs harmonics. Gabor's uncertainty relation makes this a law rather than an engineering limit. For the RMS widths of any window in time and in frequency, σt σf≥1/(4π)\sigma_t\,\sigma_f\ge 1/(4\pi), with equality only for a Gaussian.

Speech settings sit in between. A 25 ms window holds 2–3 pitch periods of a 100 Hz voice and resolves formants, and a 10 ms hop follows changes between phones. Music analysis often uses longer windows (2048 samples at 44.1 kHz = 46 ms) to resolve notes.

|STFT|² dB · fs 8 kHz · n_fft = win = 256 (32 ms) · Δf = 31.25 Hz · Hannhop 25 · frames (centred) 1 + ⌊4000/25⌋ = 161 · uncentred 1 + ⌊(4000 − 256)/25⌋ = 150 · 129 bins0 Hz100020003000400000.10.20.30.40.5 sone window, to scaletones 600 / 700 Hz (100 Hz apart = 3.2 bins): resolved, dip 11.6 dBclicks at 0.200 / 0.206 s (6 ms = 0.188 windows): merged into one smearchirp 2000 → 3800 Hz sweeps 115.2 Hz in one window · 32 ms × 31.25 Hz = 1, alwayscolour: power from −60 dB (dark) to 0 dB · marks: tones (right edge), clicks (top edge)
window shape
fig.Spectrogram of 0.5 s at 8 kHz: a 2000 → 3800 Hz chirp, tones at 600 and 700 Hz, and two clicks 6 ms apart (hop 25, centred, 161 frames). Default (Hann, 256 samples = 32 ms): Δf = 31.25 Hz, the tones are 3.2 bins apart and resolved with an 11.6 dB dip, but the clicks merge into one stripe. At 64 samples (8 ms): Δf = 125 Hz, the tones merge, and the clicks separate with a 4.9 dB dip. At 128 neither pair is resolved. At 512 the tone dip grows to 27.5 dB, while the chirp sweeps 230.4 Hz within one window.

Inversion: overlap-add

Each column can be inverted with an IFFT, which returns the windowed frame. Multiplying again by a synthesis window, adding the frames at their offsets and dividing by the summed squared window recovers xx exactly:

x^[n]=∑mw[n−mH]  ym[n−mH]∑mw2[n−mH],ym=IFFT(S[⋅,m])\hat x[n]=\frac{\sum_m w[n-mH]\;y_m[n-mH]}{\sum_m w^2[n-mH]},\qquad y_m=\mathrm{IFFT}(S[\cdot,m])
∑mw2[n−mH]\sum_m w^2[n-mH]
window envelope; must never be 0 (the NOLA condition)
COLA\text{COLA}
stronger: ∑mw[n−mH]\sum_m w[n-mH] constant. Periodic Hann sums to 1 at H=N/2H=N/2 and to 2 at H=N/4H=N/4

The speech setting 400 / 160 is not COLA for Hann: the window sum ripples between 1.19 and 1.31. It is NOLA, so torch.istft still inverts it exactly, because it divides by the envelope.

Magnitude, power and the missing phase

Models usually take the magnitude ∣S∣|S| or the power ∣S∣2|S|^2, then a mel filterbank and a log; see Mel spectrograms. In dB the two are identical, 10log⁡10∣S∣2=20log⁡10∣S∣10\log_{10}|S|^2 = 20\log_{10}|S|; with a natural log, log power is twice log magnitude. Dropping phase halves the data, and the magnitude is stable under small shifts of the frame.

To produce audio from a magnitude, a phase must be invented. Griffin–Lim starts from random phase and alternates: ISTFT to a signal, STFT again, keep the new phase, and restore the target magnitude. Each iteration does not increase ∑(∣STFT(x)∣−∣S∣)2\sum\big(|\mathrm{STFT}(x)|-|S|\big)^2. Tacotron (2017) used it as its vocoder. Its metallic, phasey artefacts are why WaveNet, WaveRNN and HiFi-GAN vocoders replaced it.

Worked example: shapes at 16 kHz, n_fft 400, hop 160

InputFrames TSpectrogramNote
1 s (16,000 samples)1 + ⌊16000/160⌋ = 101201 × 101 complex = 20,301frame rate 100 per second
1 s, center=False1 + ⌊15600/160⌋ = 98201 × 98frames end at sample 15,920; the last 80 samples are in no frame
2 s1 + 32000/160 = 201201 × 201T grows by exactly 100 per second
30 s Whisper window1 + 480000/160 = 3001Whisper drops the last: 80 mel × 3000encoder conv stride 2 → 1500
Redundancy201 complex per 160 new samples402 reals / 160 = 2.51×so not every array is a valid STFT
Frame counts and sizes (centre padding unless stated)
python
1import numpy as np
2
3def stft(x, n_fft=400, hop=160):
4 """Like torch.stft(center=True, pad_mode='reflect', window=hann_window(n_fft))."""
5 w = 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(n_fft) / n_fft) # periodic Hann
6 xp = np.pad(x, n_fft // 2, mode="reflect")
7 frames = np.lib.stride_tricks.sliding_window_view(xp, n_fft)[::hop] # (T, n_fft)
8 return np.fft.rfft(frames * w, axis=1).T, w # (n_fft/2 + 1, T)
9
10def istft(S, w, hop, length):
11 """Overlap-add with window-envelope normalisation (needs NOLA, not COLA)."""
12 n_fft = len(w)
13 frames = np.fft.irfft(S.T, n=n_fft, axis=1) * w
14 out = np.zeros(n_fft + hop * (len(frames) - 1))
15 env = np.zeros_like(out)
16 for m, f in enumerate(frames):
17 out[m * hop:m * hop + n_fft] += f
18 env[m * hop:m * hop + n_fft] += w ** 2
19 s = slice(n_fft // 2, n_fft // 2 + length) # drop the centre padding
20 return out[s] / env[s]
21
22fs = 16_000
23t = np.arange(fs) / fs
24x = np.sin(2 * np.pi * (300 * t + 1_500 * t ** 2)) # chirp 300 -> 3300 Hz in 1 s
25S, w = stft(x)
26print(S.shape) # (201, 101)
27power_db = 10 * np.log10(np.abs(S) ** 2 + 1e-10) # power spectrogram in dB
28print(power_db[:, 50].argmax() * fs / 400, "Hz") # 1800.0 at t = 0.5 s
29print(np.abs(istft(S, w, 160, len(x)) - x).max()) # ~9e-16: exact inversion
STFT and ISTFT from scratch in numpy, matching torch.stft(center=True, window=hann_window(400)) (runs as is)

Check yourself

A 2 s clip at 16 kHz goes through torch.stft(n_fft=400, hop_length=160, center=True). What is the output shape?

Check yourself

You must tell apart two clicks 6 ms apart at 8 kHz. Which window works?

Remember

  • ✓STFT: DFT of windowed frames x[n+mH]w[n]x[n+mH]w[n]; output (N/2+1)×T(N/2+1)\times T complex.
  • ✓Speech default at 16 kHz: n_fft = win = 400 (25 ms), hop 160 (10 ms), Hann → 201 bins at 40 Hz, 100 frames/s.
  • ✓Frames: 1 + ⌊L/hop⌋ with centring (N/2 padding), 1 + ⌊(L − n_fft)/hop⌋ without; Whisper's 30 s gives 3001, trimmed to 3000.
  • ✓Window length × bin spacing = 1: long windows resolve frequency, short windows resolve time (Gabor: σ_t·σ_f ≥ 1/4π).
  • ✓Overlap-add with division by Σw² inverts exactly whenever the envelope is non-zero (NOLA); Hann at hop N/2 or N/4 is also COLA.
  • ✓Magnitude or power spectrograms drop phase; Griffin–Lim (1984) or a neural vocoder must supply it to make audio.
← back to Deep Learning roadmap