Mel spectrograms

A power spectrogram pooled by triangular filters equally spaced on the mel scale m = 2595·log10(1 + f/700), then log-compressed: 80 or 128 bands × 100 frames/s is the standard input to speech and audio networks.

8 min read734 wordscore0 viewsupdated 2026-10-15

Definition

Mel spectrogram is a time–frequency image M=H ∣X∣2M=H\,|X|^2: the power spectrogram ∣X∣2|X|^2 of a Short-time Fourier transform (STFT) multiplied by a bank HH of overlapping triangular filters whose centres are equally spaced on the mel scale, a perceptual pitch scale introduced by Stevens, Volkmann and Newman (1937). Networks almost always see its logarithm, the log-mel spectrogram.

A raw STFT has too many, badly distributed rows. At 16 kHz with a 400-sample window there are 201 frequency bins, 40 Hz apart, from 0 to 8 kHz. Half of them describe 4–8 kHz, where hearing (and speech content) is coarse. Below 1 kHz, where pitch and the first formant live, the same 40 Hz spacing is barely enough.

The mel filterbank fixes the distribution: narrow filters at low frequencies, wide ones at high frequencies. It also shrinks the input from 201 to 80 numbers per frame and smooths away pitch harmonics at high frequencies. The log then compresses a dynamic range of 80–100 dB into a range a network can learn from.

The mel scale and the filterbank

mHTK(f)=2595 log⁡10 ⁣(1+f700),mSlaney(f)={3f/200f<100015+27ln⁡(f/1000)ln⁡6.4f≥1000m_{\text{HTK}}(f)=2595\,\log_{10}\!\Big(1+\frac{f}{700}\Big),\qquad m_{\text{Slaney}}(f)=\begin{cases}3f/200 & f<1000\\[2pt] 15+\dfrac{27\ln(f/1000)}{\ln 6.4} & f\ge 1000\end{cases}
mHTKm_{\text{HTK}}
the formula used by the HTK toolkit and torchaudio's default; m(1000)≈1000m(1000)\approx1000, m(8000)=2840m(8000)=2840
mSlaneym_{\text{Slaney}}
Malcolm Slaney's Auditory Toolbox version, librosa's default: exactly linear below 1 kHz, logarithmic above; m(1000)=15m(1000)=15, m(8000)=45.25m(8000)=45.25
ff
frequency in Hz. Both scales are close to linear below about 1 kHz and logarithmic above; they differ in units and in the knee

To build nn filters between fmin⁡f_{\min} and fmax⁡f_{\max}, take n+2n+2 points equally spaced in mel and convert them back to Hz. Filter kk is a triangle that rises from point kk to 1 at point k+1k+1 and falls to 0 at point k+2k+2. Neighbouring triangles overlap by half, so every FFT bin between the first and last centre is shared by two filters.

Mt,k=∑j=0nfft/2Hk,j ∣Xt,j∣2,Lt,k=log⁡(Mt,k+ε)    or    10log⁡10Mt,k floored at max⁡−80 dBM_{t,k}=\sum_{j=0}^{n_{\text{fft}}/2}H_{k,j}\,\big|X_{t,j}\big|^2,\qquad L_{t,k}=\log\big(M_{t,k}+\varepsilon\big)\;\;\text{or}\;\;10\log_{10}M_{t,k}\ \text{floored at}\ \max-80\ \text{dB}
Hk,jH_{k,j}
weight of FFT bin jj in filter kk, an nmels×(nfft/2+1)n_{\text{mels}}\times(n_{\text{fft}}/2+1) matrix, e.g. 80×20180\times201. librosa also scales each row by 2/(fk+2−fk)2/(f_{k+2}-f_k) (Slaney area normalisation); torchaudio does not by default
Xt,jX_{t,j}
STFT of frame tt at bin jj; frames of 25 ms every 10 ms are standard for speech
Lt,kL_{t,k}
log-mel feature; ε\varepsilon (e.g. 10−1010^{-10}) avoids log⁡0\log 0 in silence
1 · frame and transformx[n] · 16 kHzhop 160 (10 ms)Hann 400 samples (25 ms)|STFT|²: 201 × TH: 80 × 201 (18 drawn)08k2 · pool, compress, normalisemel poolingM = H · |X|²(80 × 201)(201 × T)→ 80 × Tlog compressionlog(M + ε)or 10·log10 M,floor at max − 80 dBnormaliseWhisper: max − 8in log10, (x+4)/4or mean/var per binlog-mel 80 × TT = 100 frames/s at a 10 ms hop: a 30 s Whisper window gives 80 × 3000 numbers per clip.3 · mel scales and real configs1 kHzHTK (torchaudio)Slaney (librosa)100 Hz · mel / mel(8 kHz) · 8 kHzmodelsrwindow / hopmelsWhisper16 kHz400 / 16080 (v3: 128)Tacotron 224 kHz50 / 12.5 ms80, 125–7600 HzHiFi-GAN V122.05 kHz1024 / 25680, ≤ 8 kHzPANNs32 kHz1024 / 32064, 50 Hz–14 kHzAST16 kHz25 / 10 ms128
fig.From waveform to log-mel. Row 1: a 16 kHz signal is cut into 400-sample Hann windows every 160 samples, giving a 201 × T power spectrogram, and an 80 × 201 matrix of triangles (18 drawn) pools it. Row 2 (right to left): mel pooling, log compression and normalisation give the 80 × T log-mel image; a 30 s Whisper input is 80 × 3000. Row 3: the HTK and Slaney scales, normalised to their value at 8 kHz, almost coincide; the table lists real front ends.

Compression and normalisation

Power spans many orders of magnitude: the synthetic vowel in the figure below covers 91 dB between its strongest and weakest mel cells. On a linear scale a few peaks dominate and everything else is near zero. A log makes equal ratios equal steps, which is also roughly how loudness is perceived. A floor (librosa's top_db=80) stops silence at log⁡ε\log\varepsilon from dominating the statistics.

Normalisation then centres the values. Whisper takes log⁡10\log_{10} of the power mel, clamps it at (max − 8), which is an 80 dB floor, and maps it with (x+4)/4(x+4)/4. Other systems subtract a per-utterance or dataset mean and divide by the standard deviation per band. Tacotron 2 clips magnitudes at a minimum of 0.01 before the log.

Real front ends

Whisper
16 kHz, n_fft 400 (25 ms), hop 160 (10 ms), 80 bins; large-v3 uses 128. 30 s windows → 80 × 3000
Tacotron 2
50 ms frames, 12.5 ms hop, Hann window, 80 bands spanning 125 Hz–7.6 kHz, log after clipping at 0.01 (Shen et al., 2018)
HiFi-GAN (V1 config)
22.05 kHz, n_fft 1024, hop 256, 80 bands up to 8 kHz; the usual LJSpeech vocoder input
PANNs
32 kHz, window 1024, hop 320, 64 bands from 50 Hz to 14 kHz (Kong et al., 2020)
AST
16 kHz, 25 ms Hamming window, 10 ms shift, 128 bands; 10 s → 128 × 1024 (Gong et al., 2021)
Library defaults
librosa: Slaney scale + area norm, n_mels=128, n_fft=2048, hop 512. torchaudio: HTK scale, no norm, n_mels=128
16 kHz · Hann 400 (25 ms) · hop 160 (10 ms) · n_fft 400 → 201 bins × 38 framesx[n]linear |STFT|² · 40 Hz per bin0k2k4k6k8kfiltersmel spectrogram · 80 bands374107821.2k1.7k2.4k3.6k5.2k7.7kframes 0 … 37 (10 ms each); warm lines join every ~10th filter centre to its mel row201 × 38 linear → 80 × 38 mel (2.51× fewer rows)lowest filter 0 Hz–74 Hz (74 Hz wide) · highest 7.41 kHz–8.00 kHz (591 Hz)centre spacing 37 Hz near 350 Hz, 230 Hz near 6 kHz · tones 300/450 resolved · 6000/6150 mergeddB, top_db 80: floor at max − 80 dB, shown range 80 dB · 28.7% of mel cells exceed 1% of the maxequal steps on the mel axis: narrow filters below 1 kHz, wide ones above, like pitch perception
signal
filterbank
compression
fig.A live log-mel spectrogram of 0.4 s of synthetic audio at 16 kHz (38 frames). Defaults (vowel /a/, librosa filters, 80 bands, 8 kHz): the lowest filter covers 0–74 Hz, the highest 7.41–8.00 kHz (591 Hz wide), and centres are 37 Hz apart near 350 Hz but 230 Hz apart near 6 kHz. With tone pairs, the linear STFT separates both 300/450 Hz and 6000/6150 Hz. The mel image separates only the low pair at 40 bands or more, and never the high pair, even at 128. With torchaudio filters and n_fft 400, 96 bands leave 1 empty filter and 128 leave 4. Switch compression to power: the raw range is 91 dB, and only 28.7% of mel cells are within 20 dB of the peak.
Scale, n_melsMel stepFirst centre / widthLast centre / widthEmpty filters
HTK, 8035.0622.1 Hz / 44.9 Hz7733.5 Hz / 524.8 Hz0
HTK, 12822.0213.8 Hz / 27.9 Hz7831.7 Hz / 333.4 Hz4
Slaney, 800.55937.2 Hz / 74.5 Hz7698.6 Hz / 591.5 Hz0
Slaney, 1280.35123.4 Hz / 46.8 Hz7809.4 Hz / 376.7 Hz0
Worked example: filterbank geometry at 16 kHz, n_fft 400 (40 Hz bins), 0–8 kHz

HTK spacing is m(8000)/(80+1)=2840.02/81=35.06m(8000)/(80+1)=2840.02/81=35.06 mel. The first HTK filter at 128 bands is 27.9 Hz wide, narrower than the 40 Hz bin spacing, so some low filters contain no bin centre and output zero forever. The width ratio last/first is about 12 for HTK-80 (524.8/44.9524.8/44.9) and 8 for Slaney-80 (591.5/74.5591.5/74.5), because Slaney stays linear up to 1 kHz.

Shapes for one 30 s Whisper window

  1. 130×16,000=480,00030\times16{,}000=480{,}000 samples; hop 160 → 480,000/160=3000480{,}000/160=3000 frames.
  2. 2Power STFT with n_fft 400: 201×3000=603,000201\times3000=603{,}000 numbers.
  3. 3Mel pooling with an 80×20180\times201 matrix: 80×3000=240,00080\times3000=240{,}000 numbers, 2.5× fewer.
  4. 4Log, 80 dB floor, (x+4)/4(x+4)/4: the encoder input, values within a range of 2.

Choosing the parameters

Window and hop set the time–frequency trade-off of the underlying STFT: 25 ms resolves harmonics of most voices, and a 10 ms hop gives 100 frames/s. fmax⁡f_{\max} is the Nyquist frequency or lower (7.6–8 kHz for TTS at 22–24 kHz, 14 kHz for PANNs at 32 kHz). A non-zero fmin⁡f_{\min} (50–125 Hz) drops hum and wasted low bands. nmelsn_{\text{mels}} must stay small enough that the lowest filters still contain FFT bins.

python
1import numpy as np
2
3def mel_filterbank(sr=16000, n_fft=400, n_mels=80, fmin=0.0, fmax=8000.0):
4 mel = lambda f: 2595 * np.log10(1 + f / 700) # HTK formula
5 hz = lambda m: 700 * (10 ** (m / 2595) - 1)
6 pts = hz(np.linspace(mel(fmin), mel(fmax), n_mels + 2)) # n_mels + 2 edges in Hz
7 f = np.arange(n_fft // 2 + 1) * sr / n_fft # FFT bin centres
8 lo, c, hi = pts[:-2, None], pts[1:-1, None], pts[2:, None]
9 return np.maximum(0, np.minimum((f - lo) / (c - lo), (hi - f) / (hi - c)))
10
11def log_mel(x, sr=16000, n_fft=400, hop=160, n_mels=80):
12 win = np.hanning(n_fft + 1)[:-1] # periodic Hann
13 frames = np.lib.stride_tricks.sliding_window_view(x, n_fft)[::hop] * win
14 power = np.abs(np.fft.rfft(frames, axis=1)) ** 2 # (T, 201)
15 mel = power @ mel_filterbank(sr, n_fft, n_mels).T # (T, 80)
16 logm = np.log10(np.maximum(mel, 1e-10))
17 logm = np.maximum(logm, logm.max() - 8.0) # Whisper: 80 dB floor
18 return ((logm + 4.0) / 4.0).T # (80, T)
19
20t = np.arange(16000) / 16000 # 1 s, 440 Hz tone
21X = log_mel(np.sin(2 * np.pi * 440 * t))
22print(X.shape, X.max(), X.min()) # (80, 98) 1.965 -0.035: range is exactly 8/4 = 2
23# librosa equivalent (Slaney scale by default):
24# librosa.feature.melspectrogram(y=x, sr=16000, n_fft=400, hop_length=160, n_mels=80)
Whisper-style log-mel in numpy (HTK filters here; Whisper itself uses librosa's Slaney filters). Run with numpy 2.5.

Use when

  • +2.5× fewer rows than a 201-bin STFT, spaced where speech information is dense
  • +Smooths high-frequency harmonics, so the image is less sensitive to pitch
  • +A fixed, differentiable linear map: cheap, and identical at training and test time

Avoid when

  • −Discards phase and fine high-frequency detail, so a vocoder is needed to get audio back
  • −Low filters can be narrower than the FFT bin spacing (empty filters) if n_fft is small
  • −Many arbitrary choices (scale, norm, f_max, log base, floor) must match the checkpoint exactly

Check yourself

With n_fft = 400 at 16 kHz, 128 HTK filters on 0–8 kHz give 4 all-zero channels. Why?

Check yourself

Whisper clamps log⁡10\log_{10} mel power at (max − 8). What does that floor equal in decibels?

Remember

  • ✓Mel spectrogram =H∣X∣2=H|X|^2: triangular filters at equal mel spacing, m=2595log⁡10(1+f/700)m=2595\log_{10}(1+f/700) (HTK) or Slaney's linear-below-1-kHz variant.
  • ✓Narrow filters at low frequency, wide at high: at 80 bands, 16 kHz, centres are 37 Hz apart near 350 Hz but 230 Hz apart near 6 kHz (Slaney).
  • ✓Always log-compress: log⁡(M+ε)\log(M+\varepsilon) or dB with an 80 dB floor; then normalise.
  • ✓Standard speech input: 25 ms window, 10 ms hop, 80 bands (Whisper, Tacotron 2, HiFi-GAN); 128 for Whisper large-v3 and AST; 64 for PANNs.
  • ✓The front end is part of the model: scale, norm, f_max, log base and normalisation must match the checkpoint.
← back to Deep Learning roadmap