# Audio signal processing

> Audio reaches a model as samples: sampled at fs (16 kHz for speech, 44.1/48 kHz for music), quantised at about 6 dB of SNR per bit, resampled, framed in 25 ms windows every 10 ms, and fed to a waveform or a spectrogram model.

Source: https://www.mlfoundry.in/learn/audio-signal-processing  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Audio signal processing:** for deep learning is the fixed chain that turns a sound wave into tensors: sampling and quantisation in the converter, then resampling, scaling, optional pre-emphasis and framing in the data loader. Its theory rests on the sampling theorem (Nyquist, 1928; Shannon, "Communication in the Presence of Noise", 1949): a signal with no energy at or above $f_s/2$ is fully determined by its samples taken $f_s$ times per second.

Sound is a small, fast variation of air pressure around the atmospheric mean. It travels at about 343 m/s at 20 °C, and humans hear roughly 20 Hz to 20 kHz. A microphone turns pressure into a voltage. An analogue-to-digital converter (ADC) turns the voltage into a sequence of integers. Every choice made here (sample rate, bit depth, filters) limits what any later model can learn.

The model never sees "sound". It sees a 1-D array of about 16,000 numbers per second of speech, or 44,100 to 48,000 for music. It either learns directly from that array, or from a time–frequency image computed from it.

## Sampling and aliasing

Sampling keeps the values $x[n]=x(n/f_s)$. The **Nyquist frequency** is $f_s/2$. Any tone above it produces exactly the same samples as a tone inside $[0, f_s/2]$, so after sampling the two cannot be told apart. This is **aliasing**:

$$
f_{\text{alias}} = \big|\,f - f_s\cdot\operatorname{round}(f/f_s)\,\big|\;\in\;[0,\ f_s/2]
$$

- $f$: frequency of the analogue tone
- $f_s$: sample rate; $f_s/2$ is the Nyquist frequency
- $f_{\text{alias}}$: the frequency the samples actually represent; e.g. 15 kHz at $f_s=16$ kHz → 1 kHz

Aliasing cannot be undone after sampling. Converters therefore apply an **anti-aliasing low-pass filter** before the ADC, and every resampler applies one before it reduces the rate. Telephony runs at 8 kHz with a 300–3400 Hz band. Speech recognition uses 16 kHz, which keeps everything below 8 kHz, where almost all speech information lies. TTS models often use 22.05 or 24 kHz. Music uses 44.1 kHz (CD) or 48 kHz (video and studio), just above twice the 20 kHz hearing limit.

> Figure: From sound to model input. Top: pressure is low-pass filtered below fs/2, then sampled and quantised; 16-bit PCM at 16 kHz is 256 kbit/s, CD stereo is 1411.2 kbit/s. Middle: the data loader resamples (44.1 → 16 kHz is the rational ratio 160/441), scales int16 to [−1, 1), optionally pre-emphasises, and frames into 25 ms windows every 10 ms. Bottom: a waveform model (wav2vec 2.0's 7-layer CNN, total stride 320) or a spectrogram model on 80 log-mel bands.

## Quantisation, bit depth and decibels

Each sample is rounded to one of $2^b$ levels. The rounding error behaves like white noise of power $\Delta^2/12$ for a step $\Delta$. For a full-scale sine this gives the rule of thumb of about 6 dB per bit:

$$
\mathrm{SQNR} \approx 6.02\,b + 1.76\ \text{dB}\qquad(b=16:\ 98.1\ \text{dB};\ \ b=24:\ 146.2\ \text{dB})
$$

- $b$: bits per sample; $6.02 = 20\log_{10}2$, one bit doubles the number of levels
- $1.76$: $10\log_{10}(3/2)$: ratio of a full-scale sine's power to the noise power $\Delta^2/12$, beyond the $2^{2b}$ term

Levels are given in decibels: $20\log_{10}$ of an amplitude ratio, or $10\log_{10}$ of a power ratio. In software the reference is full scale (dBFS, where 0 dBFS is the largest sample). In air it is 20 µPa (dB SPL). Perceived loudness is measured in LUFS (ITU-R BS.1770); broadcast normalises to −23 LUFS under EBU R128. A signal at −6 dBFS uses half the range and loses 6 dB of SQNR.

Some systems spend bits non-uniformly. G.711 telephony uses 8-bit μ-law ($\mu=255$) at 8 kHz, which gives 64 kbit/s. **WaveNet** predicts 8-bit μ-law values as a 256-way softmax.

**Common formats**

- **Telephony:** 8 kHz, 8-bit μ-law or A-law (G.711), 64 kbit/s, band 300–3400 Hz
- **Speech models:** 16 kHz mono: wav2vec 2.0, HuBERT, Whisper; 16-bit PCM is 256 kbit/s
- **TTS:** 22.05 kHz (LJSpeech, HiFi-GAN configs) or 24 kHz
- **CD audio:** 44.1 kHz, 16-bit, stereo: 1411.2 kbit/s
- **Video / studio:** 48 kHz (and multiples), 24-bit
- **Float audio:** libraries load samples as float32 in [−1, 1): int16 divided by 32768

> Figure: A full-scale sine sampled and quantised, with the SQNR measured over 4096 samples. Default (1.1 kHz, fs = 16 kHz, 4-bit): 14.55 samples per period, measured SQNR 25.3 dB against the rule's 25.8 dB. Raise the tone to 15 kHz: the 33 samples in 2 ms fall exactly on a sign-flipped 1 kHz sine (dashed). 9 kHz folds to 7 kHz. At 16 bits the measured SQNR is 98.2 dB against the rule's 98.1 dB. Switch to 8 kHz and set f = 12 kHz: every sample is 0.

## Preprocessing in the data loader

**Resampling.** A model trained at 16 kHz must get 16 kHz input. Rational resampling by $L/M$ upsamples by $L$, low-pass filters below the lower Nyquist, and keeps every $M$-th sample. Polyphase filters avoid computing the discarded samples. 44.1 → 16 kHz is $L/M = 160/441$, since $\gcd(44100, 16000) = 100$.

**Scaling and channels.** Stereo is usually averaged to mono. int16 is divided by 32768. Datasets often peak-normalise or loudness-normalise clips, so that level is not a shortcut feature.

**Pre-emphasis.** Voiced speech loses energy with frequency. A first-order high-pass boosts the highs before analysis:

$$
y[n] = x[n] - \alpha\,x[n-1],\qquad |H(f)| = \big|1-\alpha\,e^{-j2\pi f/f_s}\big|
$$

- $\alpha$: typically 0.95–0.97; at $\alpha=0.97$ and 16 kHz the gain is −30.5 dB at DC, −8.3 dB at 1 kHz, +2.9 dB at 4 kHz and +5.9 dB at 8 kHz

Pre-emphasis belongs to the classic **MFCC** front end and to some TTS recipes. Whisper and most learned front ends skip it.

**Framing.** Speech is roughly stationary over 10–30 ms, so analysis uses overlapping frames: 25 ms windows every 10 ms (400 and 160 samples at 16 kHz), or 100 frames per second. Each frame is windowed and transformed, see **Short-time Fourier transform (STFT)**.

## Worked example: a 10 s music clip to model inputs

Sizes at each stage for a 10 s, 44.1 kHz, 16-bit stereo clip

| Stage | Computation | Result |
| --- | --- | --- |
| Raw PCM | 441,000 samples × 2 channels × 2 bytes | 1,764,000 bytes |
| Mono, resampled to 16 kHz | 10 × 16,000 samples, float32 | 160,000 samples = 640,000 bytes |
| Frames, no padding | 1 + ⌊(160,000 − 400)/160⌋ | 998 frames of 400 |
| Log-mel (centred STFT) | (1 + 160,000/160) frames × 80 bands | 80 × 1001 = 80,080 values |
| wav2vec 2.0 CNN | 7 convs, kernels 10,3,3,3,3,2,2, strides 5,2,2,2,2,2,2 | 499 frames (stride 320 = 20 ms, field 400 = 25 ms) |

```python
import numpy as np

fs = 16_000
t = np.arange(fs) / fs                                   # 1 s
x = 0.5 * np.sin(2 * np.pi * 220 * t) + 0.01 * np.random.default_rng(0).standard_normal(fs)

# Quantise to int16 PCM and back, as a WAV file would
pcm = np.clip(np.round(x * 32768), -32768, 32767).astype(np.int16)
y = pcm.astype(np.float32) / 32768.0
print("int16 round-trip SNR:", round(10 * np.log10(np.sum(x**2) / np.sum((y - x)**2)), 1), "dB")
# 92.1 dB: the 98.1 dB of full scale minus 6 dB, since the signal uses half the range

# Aliasing: a 15 kHz tone sampled at 16 kHz gives the same samples as a 1 kHz tone (sign flipped)
n = np.arange(64)
print(np.allclose(np.sin(2*np.pi*15_000*n/fs), -np.sin(2*np.pi*1_000*n/fs)))   # True

# Peak level in dBFS and pre-emphasis
print("peak:", round(20 * np.log10(np.abs(y).max()), 2), "dBFS")                # -5.5 dBFS
y_pre = np.append(y[0], y[1:] - 0.97 * y[:-1])

# Framing: 25 ms windows every 10 ms -> (frames, 400)
win, hop = int(0.025 * fs), int(0.010 * fs)
frames = np.lib.stride_tricks.sliding_window_view(y_pre, win)[::hop]
print("frames:", frames.shape)                           # (98, 400) = 1 + (16000 - 400) // 160
```

PCM round trip, aliasing, dBFS, pre-emphasis and framing in numpy (runs as is; for resampling use torchaudio.functional.resample or scipy.signal.resample_poly)

## Waveform or spectrogram input?

A **spectrogram model** starts from a fixed transform: STFT magnitude, mel filterbank, log (see **Mel spectrograms**). The input is a compact 80 × 100 image per second that discards phase. Whisper, Conformer recognisers and most audio classifiers work this way. A **waveform model** learns its front end. **wav2vec 2.0** and **HuBERT** use a strided CNN that reduces 16,000 samples per second to 50 vectors. Generators such as **WaveNet** and **HiFi-GAN** must output waveforms, so they operate at the sample rate.

**Strengths**

- Spectrogram input: compact, matches perception, cheap to augment (SpecAugment); a sound default for recognition and classification.
- Waveform input: nothing discarded (phase, fine timing); the front end is learned end to end; required for generation.

**Limitations**

- Spectrogram input: phase is lost, so generating audio needs a vocoder; front-end settings must match between training and inference.
- Waveform input: 16k–48k steps per second; costly at the sample rate and sensitive to the level and DC offset.

> **Sample-rate mismatch fails silently:** A 44.1 kHz file fed to a 16 kHz model without resampling is read as audio 2.76 times longer and lower in pitch ($44100/16000 = 2.756$). Nothing crashes; accuracy just collapses. Check the rate on load (`torchaudio.load` and `soundfile.read` both return it) and resample once, with an anti-aliasing filter.

**Question:** A 15 kHz tone reaches a 16 kHz ADC whose anti-aliasing filter is missing. What do the samples contain?

- A. Nothing: the tone is above Nyquist and is dropped
- B. A 15 kHz tone
- C. A 1 kHz tone (sign-flipped)
- D. A 7 kHz tone

**Answer:** C. A 1 kHz tone (sign-flipped) — $f_{\text{alias}} = |15 - 16\cdot\operatorname{round}(15/16)| = |15-16| = 1$ kHz. Because $f - f_s < 0$, the spectrum is mirrored and the samples equal $-\sin(2\pi\cdot 1\,\text{kHz}\cdot n/f_s)$. Only a filter before sampling can remove it.

**Question:** Going from 16-bit to 24-bit samples raises the quantisation SNR of a full-scale sine by about how much?

- A. 8 dB
- B. 24 dB
- C. 48 dB
- D. 50%

**Answer:** C. 48 dB — About 6.02 dB per bit: $8 \times 6.02 = 48.2$ dB, from 98.1 dB to 146.2 dB. Real converters stop well short of 146 dB because analogue noise dominates.

**Key points**

- Sampling at $f_s$ captures everything below $f_s/2$; anything above aliases to $|f - f_s\,\mathrm{round}(f/f_s)|$ and must be filtered out first.
- 16 kHz for speech models, 22.05/24 kHz for TTS, 44.1/48 kHz for music; 8 kHz telephony keeps only 300–3400 Hz.
- Quantisation SQNR ≈ 6.02·b + 1.76 dB: 98 dB at 16 bits; level in dBFS, loudness in LUFS.
- Loader chain: resample (polyphase, e.g. 160/441), mono, int16 / 32768, optional pre-emphasis (α ≈ 0.97), 25 ms / 10 ms frames.
- Waveform models learn the front end (wav2vec 2.0: stride 320, 50 frames/s); spectrogram models start from log-mel (80 × 100 per second).
- Match sample rate and front-end settings between training and inference; a mismatch fails silently.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
