# Mel spectrograms

> A power spectrogram pooled by triangular filters equally spaced on the mel scale m = 2595·log10(1 + f/700), then log-compressed: 80 or 128 bands × 100 frames/s is the standard input to speech and audio networks.

Source: https://www.mlfoundry.in/learn/mel-spectrograms  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Mel spectrogram:** is a time–frequency image $M=H\,|X|^2$: the power spectrogram $|X|^2$ of a **Short-time Fourier transform (STFT)** multiplied by a bank $H$ of overlapping triangular filters whose centres are equally spaced on the **mel scale**, a perceptual pitch scale introduced by Stevens, Volkmann and Newman (1937). Networks almost always see its logarithm, the **log-mel spectrogram**.

A raw STFT has too many, badly distributed rows. At 16 kHz with a 400-sample window there are 201 frequency bins, 40 Hz apart, from 0 to 8 kHz. Half of them describe 4–8 kHz, where hearing (and speech content) is coarse. Below 1 kHz, where pitch and the first formant live, the same 40 Hz spacing is barely enough.

The mel filterbank fixes the distribution: narrow filters at low frequencies, wide ones at high frequencies. It also shrinks the input from 201 to 80 numbers per frame and smooths away pitch harmonics at high frequencies. The log then compresses a dynamic range of 80–100 dB into a range a network can learn from.

## The mel scale and the filterbank

$$
m_{\text{HTK}}(f)=2595\,\log_{10}\!\Big(1+\frac{f}{700}\Big),\qquad m_{\text{Slaney}}(f)=\begin{cases}3f/200 & f<1000\\[2pt] 15+\dfrac{27\ln(f/1000)}{\ln 6.4} & f\ge 1000\end{cases}
$$

- $m_{\text{HTK}}$: the formula used by the HTK toolkit and torchaudio's default; $m(1000)\approx1000$, $m(8000)=2840$
- $m_{\text{Slaney}}$: Malcolm Slaney's Auditory Toolbox version, librosa's default: exactly linear below 1 kHz, logarithmic above; $m(1000)=15$, $m(8000)=45.25$
- $f$: frequency in Hz. Both scales are close to linear below about 1 kHz and logarithmic above; they differ in units and in the knee

To build $n$ filters between $f_{\min}$ and $f_{\max}$, take $n+2$ points equally spaced in mel and convert them back to Hz. Filter $k$ is a triangle that rises from point $k$ to 1 at point $k+1$ and falls to 0 at point $k+2$. Neighbouring triangles overlap by half, so every FFT bin between the first and last centre is shared by two filters.

$$
M_{t,k}=\sum_{j=0}^{n_{\text{fft}}/2}H_{k,j}\,\big|X_{t,j}\big|^2,\qquad L_{t,k}=\log\big(M_{t,k}+\varepsilon\big)\;\;\text{or}\;\;10\log_{10}M_{t,k}\ \text{floored at}\ \max-80\ \text{dB}
$$

- $H_{k,j}$: weight of FFT bin $j$ in filter $k$, an $n_{\text{mels}}\times(n_{\text{fft}}/2+1)$ matrix, e.g. $80\times201$. librosa also scales each row by $2/(f_{k+2}-f_k)$ (Slaney area normalisation); torchaudio does not by default
- $X_{t,j}$: STFT of frame $t$ at bin $j$; frames of 25 ms every 10 ms are standard for speech
- $L_{t,k}$: log-mel feature; $\varepsilon$ (e.g. $10^{-10}$) avoids $\log 0$ in silence

> Figure: From waveform to log-mel. Row 1: a 16 kHz signal is cut into 400-sample Hann windows every 160 samples, giving a 201 × T power spectrogram, and an 80 × 201 matrix of triangles (18 drawn) pools it. Row 2 (right to left): mel pooling, log compression and normalisation give the 80 × T log-mel image; a 30 s Whisper input is 80 × 3000. Row 3: the HTK and Slaney scales, normalised to their value at 8 kHz, almost coincide; the table lists real front ends.

## Compression and normalisation

Power spans many orders of magnitude: the synthetic vowel in the figure below covers 91 dB between its strongest and weakest mel cells. On a linear scale a few peaks dominate and everything else is near zero. A log makes equal ratios equal steps, which is also roughly how loudness is perceived. A floor (librosa's `top_db=80`) stops silence at $\log\varepsilon$ from dominating the statistics.

Normalisation then centres the values. Whisper takes $\log_{10}$ of the power mel, clamps it at (max − 8), which is an 80 dB floor, and maps it with $(x+4)/4$. Other systems subtract a per-utterance or dataset mean and divide by the standard deviation per band. Tacotron 2 clips magnitudes at a minimum of 0.01 before the log.

**Real front ends**

- **Whisper:** 16 kHz, `n_fft` 400 (25 ms), hop 160 (10 ms), 80 bins; large-v3 uses 128. 30 s windows → 80 × 3000
- **Tacotron 2:** 50 ms frames, 12.5 ms hop, Hann window, 80 bands spanning 125 Hz–7.6 kHz, log after clipping at 0.01 (Shen et al., 2018)
- **HiFi-GAN (V1 config):** 22.05 kHz, `n_fft` 1024, hop 256, 80 bands up to 8 kHz; the usual LJSpeech vocoder input
- **PANNs:** 32 kHz, window 1024, hop 320, 64 bands from 50 Hz to 14 kHz (Kong et al., 2020)
- **AST:** 16 kHz, 25 ms Hamming window, 10 ms shift, 128 bands; 10 s → 128 × 1024 (Gong et al., 2021)
- **Library defaults:** librosa: Slaney scale + area norm, `n_mels`=128, `n_fft`=2048, hop 512. torchaudio: HTK scale, no norm, `n_mels`=128

> Figure: A live log-mel spectrogram of 0.4 s of synthetic audio at 16 kHz (38 frames). Defaults (vowel /a/, librosa filters, 80 bands, 8 kHz): the lowest filter covers 0–74 Hz, the highest 7.41–8.00 kHz (591 Hz wide), and centres are 37 Hz apart near 350 Hz but 230 Hz apart near 6 kHz. With tone pairs, the linear STFT separates both 300/450 Hz and 6000/6150 Hz. The mel image separates only the low pair at 40 bands or more, and never the high pair, even at 128. With torchaudio filters and n_fft 400, 96 bands leave 1 empty filter and 128 leave 4. Switch compression to power: the raw range is 91 dB, and only 28.7% of mel cells are within 20 dB of the peak.

Worked example: filterbank geometry at 16 kHz, n_fft 400 (40 Hz bins), 0–8 kHz

| Scale, n_mels | Mel step | First centre / width | Last centre / width | Empty filters |
| --- | --- | --- | --- | --- |
| HTK, 80 | 35.06 | 22.1 Hz / 44.9 Hz | 7733.5 Hz / 524.8 Hz | 0 |
| HTK, 128 | 22.02 | 13.8 Hz / 27.9 Hz | 7831.7 Hz / 333.4 Hz | 4 |
| Slaney, 80 | 0.559 | 37.2 Hz / 74.5 Hz | 7698.6 Hz / 591.5 Hz | 0 |
| Slaney, 128 | 0.351 | 23.4 Hz / 46.8 Hz | 7809.4 Hz / 376.7 Hz | 0 |

HTK spacing is $m(8000)/(80+1)=2840.02/81=35.06$ mel. The first HTK filter at 128 bands is 27.9 Hz wide, narrower than the 40 Hz bin spacing, so some low filters contain no bin centre and output zero forever. The width ratio last/first is about 12 for HTK-80 ($524.8/44.9$) and 8 for Slaney-80 ($591.5/74.5$), because Slaney stays linear up to 1 kHz.

**Shapes for one 30 s Whisper window**

1. $30\times16{,}000=480{,}000$ samples; hop 160 → $480{,}000/160=3000$ frames.
2. Power STFT with `n_fft` 400: $201\times3000=603{,}000$ numbers.
3. Mel pooling with an $80\times201$ matrix: $80\times3000=240{,}000$ numbers, 2.5× fewer.
4. Log, 80 dB floor, $(x+4)/4$: the encoder input, values within a range of 2.

## Choosing the parameters

**Window and hop** set the time–frequency trade-off of the underlying STFT: 25 ms resolves harmonics of most voices, and a 10 ms hop gives 100 frames/s. **$f_{\max}$** is the Nyquist frequency or lower (7.6–8 kHz for TTS at 22–24 kHz, 14 kHz for PANNs at 32 kHz). A non-zero **$f_{\min}$** (50–125 Hz) drops hum and wasted low bands. **$n_{\text{mels}}$** must stay small enough that the lowest filters still contain FFT bins.

```python
import numpy as np

def mel_filterbank(sr=16000, n_fft=400, n_mels=80, fmin=0.0, fmax=8000.0):
    mel = lambda f: 2595 * np.log10(1 + f / 700)            # HTK formula
    hz = lambda m: 700 * (10 ** (m / 2595) - 1)
    pts = hz(np.linspace(mel(fmin), mel(fmax), n_mels + 2))  # n_mels + 2 edges in Hz
    f = np.arange(n_fft // 2 + 1) * sr / n_fft               # FFT bin centres
    lo, c, hi = pts[:-2, None], pts[1:-1, None], pts[2:, None]
    return np.maximum(0, np.minimum((f - lo) / (c - lo), (hi - f) / (hi - c)))

def log_mel(x, sr=16000, n_fft=400, hop=160, n_mels=80):
    win = np.hanning(n_fft + 1)[:-1]                         # periodic Hann
    frames = np.lib.stride_tricks.sliding_window_view(x, n_fft)[::hop] * win
    power = np.abs(np.fft.rfft(frames, axis=1)) ** 2         # (T, 201)
    mel = power @ mel_filterbank(sr, n_fft, n_mels).T        # (T, 80)
    logm = np.log10(np.maximum(mel, 1e-10))
    logm = np.maximum(logm, logm.max() - 8.0)                # Whisper: 80 dB floor
    return ((logm + 4.0) / 4.0).T                            # (80, T)

t = np.arange(16000) / 16000                                 # 1 s, 440 Hz tone
X = log_mel(np.sin(2 * np.pi * 440 * t))
print(X.shape, X.max(), X.min())   # (80, 98) 1.965 -0.035: range is exactly 8/4 = 2
# librosa equivalent (Slaney scale by default):
# librosa.feature.melspectrogram(y=x, sr=16000, n_fft=400, hop_length=160, n_mels=80)
```

Whisper-style log-mel in numpy (HTK filters here; Whisper itself uses librosa's Slaney filters). Run with numpy 2.5.

> **Filterbanks are not interchangeable:** librosa (Slaney, area-normalised) and torchaudio (HTK, unnormalised) give different matrices for the same `n_mels`. Feeding a model features from the wrong one, or with a different `f_max`, log base, floor or normalisation, shifts every input value. Pretrained checkpoints are only valid with the exact front end they were trained on. Copy it from the model's own preprocessing code.

> **Why 80 bands:** 80 bands at 16–24 kHz are dense enough to keep formants and the low harmonics that carry pitch, and coarse enough that adjacent frames form a compact image. Recognisers need less detail (classic systems used about 23–40), and vocoders need more, because they must rebuild the waveform from it. 128 bands (Whisper large-v3, AST) add resolution at little cost.

**Strengths**

- 2.5× fewer rows than a 201-bin STFT, spaced where speech information is dense
- Smooths high-frequency harmonics, so the image is less sensitive to pitch
- A fixed, differentiable linear map: cheap, and identical at training and test time

**Limitations**

- Discards phase and fine high-frequency detail, so a vocoder is needed to get audio back
- Low filters can be narrower than the FFT bin spacing (empty filters) if n_fft is small
- Many arbitrary choices (scale, norm, f_max, log base, floor) must match the checkpoint exactly

**Question:** With `n_fft` = 400 at 16 kHz, 128 HTK filters on 0–8 kHz give 4 all-zero channels. Why?

- A. The log of zero is undefined
- B. The lowest filters are narrower than the 40 Hz FFT bin spacing, so no bin centre falls inside them
- C. Frequencies above 8 kHz alias into the low bands
- D. HTK filters are area-normalised

**Answer:** B. The lowest filters are narrower than the 40 Hz FFT bin spacing, so no bin centre falls inside them — Bin spacing is $16000/400=40$ Hz, but the first HTK-128 filter is only 27.9 Hz wide. A triangle that contains no bin centre has all-zero weights. Fix it with a larger `n_fft` (zero-padding), fewer bands or a higher $f_{\min}$.

**Question:** Whisper clamps $\log_{10}$ mel power at (max − 8). What does that floor equal in decibels?

- A. 8 dB
- B. 40 dB
- C. 80 dB
- D. 160 dB

**Answer:** C. 80 dB — A power ratio of $10^{8}$ is $10\log_{10}10^{8}=80$ dB, the same floor as librosa's `top_db=80`. After $(x+4)/4$ the values span at most $8/4=2$ units.

**Key points**

- Mel spectrogram $=H|X|^2$: triangular filters at equal mel spacing, $m=2595\log_{10}(1+f/700)$ (HTK) or Slaney's linear-below-1-kHz variant.
- Narrow filters at low frequency, wide at high: at 80 bands, 16 kHz, centres are 37 Hz apart near 350 Hz but 230 Hz apart near 6 kHz (Slaney).
- Always log-compress: $\log(M+\varepsilon)$ or dB with an 80 dB floor; then normalise.
- Standard speech input: 25 ms window, 10 ms hop, 80 bands (Whisper, Tacotron 2, HiFi-GAN); 128 for Whisper large-v3 and AST; 64 for PANNs.
- The front end is part of the model: scale, norm, f_max, log base and normalisation must match the checkpoint.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
