Definition
Mel spectrogram is a time–frequency image : the power spectrogram of a Short-time Fourier transform (STFT) multiplied by a bank of overlapping triangular filters whose centres are equally spaced on the mel scale, a perceptual pitch scale introduced by Stevens, Volkmann and Newman (1937). Networks almost always see its logarithm, the log-mel spectrogram.
A raw STFT has too many, badly distributed rows. At 16 kHz with a 400-sample window there are 201 frequency bins, 40 Hz apart, from 0 to 8 kHz. Half of them describe 4–8 kHz, where hearing (and speech content) is coarse. Below 1 kHz, where pitch and the first formant live, the same 40 Hz spacing is barely enough.
The mel filterbank fixes the distribution: narrow filters at low frequencies, wide ones at high frequencies. It also shrinks the input from 201 to 80 numbers per frame and smooths away pitch harmonics at high frequencies. The log then compresses a dynamic range of 80–100 dB into a range a network can learn from.
The mel scale and the filterbank
- the formula used by the HTK toolkit and torchaudio's default; ,
- Malcolm Slaney's Auditory Toolbox version, librosa's default: exactly linear below 1 kHz, logarithmic above; ,
- frequency in Hz. Both scales are close to linear below about 1 kHz and logarithmic above; they differ in units and in the knee
To build filters between and , take points equally spaced in mel and convert them back to Hz. Filter is a triangle that rises from point to 1 at point and falls to 0 at point . Neighbouring triangles overlap by half, so every FFT bin between the first and last centre is shared by two filters.
- weight of FFT bin in filter , an matrix, e.g. . librosa also scales each row by (Slaney area normalisation); torchaudio does not by default
- STFT of frame at bin ; frames of 25 ms every 10 ms are standard for speech
- log-mel feature; (e.g. ) avoids in silence
Compression and normalisation
Power spans many orders of magnitude: the synthetic vowel in the figure below covers 91 dB between its strongest and weakest mel cells. On a linear scale a few peaks dominate and everything else is near zero. A log makes equal ratios equal steps, which is also roughly how loudness is perceived. A floor (librosa's top_db=80) stops silence at from dominating the statistics.
Normalisation then centres the values. Whisper takes of the power mel, clamps it at (max − 8), which is an 80 dB floor, and maps it with . Other systems subtract a per-utterance or dataset mean and divide by the standard deviation per band. Tacotron 2 clips magnitudes at a minimum of 0.01 before the log.
Real front ends
- Whisper
- 16 kHz,
n_fft400 (25 ms), hop 160 (10 ms), 80 bins; large-v3 uses 128. 30 s windows → 80 × 3000 - Tacotron 2
- 50 ms frames, 12.5 ms hop, Hann window, 80 bands spanning 125 Hz–7.6 kHz, log after clipping at 0.01 (Shen et al., 2018)
- HiFi-GAN (V1 config)
- 22.05 kHz,
n_fft1024, hop 256, 80 bands up to 8 kHz; the usual LJSpeech vocoder input - PANNs
- 32 kHz, window 1024, hop 320, 64 bands from 50 Hz to 14 kHz (Kong et al., 2020)
- AST
- 16 kHz, 25 ms Hamming window, 10 ms shift, 128 bands; 10 s → 128 × 1024 (Gong et al., 2021)
- Library defaults
- librosa: Slaney scale + area norm,
n_mels=128,n_fft=2048, hop 512. torchaudio: HTK scale, no norm,n_mels=128
| Scale, n_mels | Mel step | First centre / width | Last centre / width | Empty filters |
|---|---|---|---|---|
| HTK, 80 | 35.06 | 22.1 Hz / 44.9 Hz | 7733.5 Hz / 524.8 Hz | 0 |
| HTK, 128 | 22.02 | 13.8 Hz / 27.9 Hz | 7831.7 Hz / 333.4 Hz | 4 |
| Slaney, 80 | 0.559 | 37.2 Hz / 74.5 Hz | 7698.6 Hz / 591.5 Hz | 0 |
| Slaney, 128 | 0.351 | 23.4 Hz / 46.8 Hz | 7809.4 Hz / 376.7 Hz | 0 |
HTK spacing is mel. The first HTK filter at 128 bands is 27.9 Hz wide, narrower than the 40 Hz bin spacing, so some low filters contain no bin centre and output zero forever. The width ratio last/first is about 12 for HTK-80 () and 8 for Slaney-80 (), because Slaney stays linear up to 1 kHz.
Shapes for one 30 s Whisper window
- 1 samples; hop 160 → frames.
- 2Power STFT with
n_fft400: numbers. - 3Mel pooling with an matrix: numbers, 2.5× fewer.
- 4Log, 80 dB floor, : the encoder input, values within a range of 2.
Choosing the parameters
Window and hop set the time–frequency trade-off of the underlying STFT: 25 ms resolves harmonics of most voices, and a 10 ms hop gives 100 frames/s. is the Nyquist frequency or lower (7.6–8 kHz for TTS at 22–24 kHz, 14 kHz for PANNs at 32 kHz). A non-zero (50–125 Hz) drops hum and wasted low bands. must stay small enough that the lowest filters still contain FFT bins.
1import numpy as np23def mel_filterbank(sr=16000, n_fft=400, n_mels=80, fmin=0.0, fmax=8000.0):4 mel = lambda f: 2595 * np.log10(1 + f / 700) # HTK formula5 hz = lambda m: 700 * (10 ** (m / 2595) - 1)6 pts = hz(np.linspace(mel(fmin), mel(fmax), n_mels + 2)) # n_mels + 2 edges in Hz7 f = np.arange(n_fft // 2 + 1) * sr / n_fft # FFT bin centres8 lo, c, hi = pts[:-2, None], pts[1:-1, None], pts[2:, None]9 return np.maximum(0, np.minimum((f - lo) / (c - lo), (hi - f) / (hi - c)))1011def log_mel(x, sr=16000, n_fft=400, hop=160, n_mels=80):12 win = np.hanning(n_fft + 1)[:-1] # periodic Hann13 frames = np.lib.stride_tricks.sliding_window_view(x, n_fft)[::hop] * win14 power = np.abs(np.fft.rfft(frames, axis=1)) ** 2 # (T, 201)15 mel = power @ mel_filterbank(sr, n_fft, n_mels).T # (T, 80)16 logm = np.log10(np.maximum(mel, 1e-10))17 logm = np.maximum(logm, logm.max() - 8.0) # Whisper: 80 dB floor18 return ((logm + 4.0) / 4.0).T # (80, T)1920t = np.arange(16000) / 16000 # 1 s, 440 Hz tone21X = log_mel(np.sin(2 * np.pi * 440 * t))22print(X.shape, X.max(), X.min()) # (80, 98) 1.965 -0.035: range is exactly 8/4 = 223# librosa equivalent (Slaney scale by default):24# librosa.feature.melspectrogram(y=x, sr=16000, n_fft=400, hop_length=160, n_mels=80)Use when
- +2.5× fewer rows than a 201-bin STFT, spaced where speech information is dense
- +Smooths high-frequency harmonics, so the image is less sensitive to pitch
- +A fixed, differentiable linear map: cheap, and identical at training and test time
Avoid when
- −Discards phase and fine high-frequency detail, so a vocoder is needed to get audio back
- −Low filters can be narrower than the FFT bin spacing (empty filters) if n_fft is small
- −Many arbitrary choices (scale, norm, f_max, log base, floor) must match the checkpoint exactly
Check yourself
With n_fft = 400 at 16 kHz, 128 HTK filters on 0–8 kHz give 4 all-zero channels. Why?
Check yourself
Whisper clamps mel power at (max − 8). What does that floor equal in decibels?
Remember
- ✓Mel spectrogram : triangular filters at equal mel spacing, (HTK) or Slaney's linear-below-1-kHz variant.
- ✓Narrow filters at low frequency, wide at high: at 80 bands, 16 kHz, centres are 37 Hz apart near 350 Hz but 230 Hz apart near 6 kHz (Slaney).
- ✓Always log-compress: or dB with an 80 dB floor; then normalise.
- ✓Standard speech input: 25 ms window, 10 ms hop, 80 bands (Whisper, Tacotron 2, HiFi-GAN); 128 for Whisper large-v3 and AST; 64 for PANNs.
- ✓The front end is part of the model: scale, norm, f_max, log base and normalisation must match the checkpoint.