Audio signal processing

Audio reaches a model as samples: sampled at fs (16 kHz for speech, 44.1/48 kHz for music), quantised at about 6 dB of SNR per bit, resampled, framed in 25 ms windows every 10 ms, and fed to a waveform or a spectrogram model.

8 min read826 wordscore0 viewsupdated 2026-10-15

Definition

Audio signal processing for deep learning is the fixed chain that turns a sound wave into tensors: sampling and quantisation in the converter, then resampling, scaling, optional pre-emphasis and framing in the data loader. Its theory rests on the sampling theorem (Nyquist, 1928; Shannon, "Communication in the Presence of Noise", 1949): a signal with no energy at or above fs/2f_s/2 is fully determined by its samples taken fsf_s times per second.

Sound is a small, fast variation of air pressure around the atmospheric mean. It travels at about 343 m/s at 20 °C, and humans hear roughly 20 Hz to 20 kHz. A microphone turns pressure into a voltage. An analogue-to-digital converter (ADC) turns the voltage into a sequence of integers. Every choice made here (sample rate, bit depth, filters) limits what any later model can learn.

The model never sees "sound". It sees a 1-D array of about 16,000 numbers per second of speech, or 44,100 to 48,000 for music. It either learns directly from that array, or from a time–frequency image computed from it.

Sampling and aliasing

Sampling keeps the values x[n]=x(n/fs)x[n]=x(n/f_s). The Nyquist frequency is fs/2f_s/2. Any tone above it produces exactly the same samples as a tone inside [0,fs/2][0, f_s/2], so after sampling the two cannot be told apart. This is aliasing:

falias=∣ f−fs⋅round⁡(f/fs) ∣  ∈  [0, fs/2]f_{\text{alias}} = \big|\,f - f_s\cdot\operatorname{round}(f/f_s)\,\big|\;\in\;[0,\ f_s/2]
ff
frequency of the analogue tone
fsf_s
sample rate; fs/2f_s/2 is the Nyquist frequency
faliasf_{\text{alias}}
the frequency the samples actually represent; e.g. 15 kHz at fs=16f_s=16 kHz → 1 kHz

Aliasing cannot be undone after sampling. Converters therefore apply an anti-aliasing low-pass filter before the ADC, and every resampler applies one before it reduces the rate. Telephony runs at 8 kHz with a 300–3400 Hz band. Speech recognition uses 16 kHz, which keeps everything below 8 kHz, where almost all speech information lies. TTS models often use 22.05 or 24 kHz. Music uses 44.1 kHz (CD) or 48 kHz (video and studio), just above twice the 20 kHz hearing limit.

1 · analogue → digital (in the microphone / sound card)pressure p(t)20 Hz – 20 kHz hearinganti-alias LPFremove energyabove fs/2(else it foldsinto the band)ADC: sample + quantiseevery 1/fs · 2^b levelsPCM samplesint16: 96 dB range16 kHz → 256 kbit/s44.1 kHz stereo→ 1411.2 kbit/s2 · digital preprocessing (in the data loader)resample44.1 → 16 kHzratio 160/441polyphase LPFmono + scaleint16 / 32768→ [−1, 1) floatpeak / LUFS normpre-emphasisy[n] = x[n]− 0.97·x[n−1](MFCC, some TTS)frame25 ms = 400hop 10 ms = 160100 frames / s3 · model inputwaveform modelinput 16,000 samples per secondwav2vec 2.0 / HuBERT: 7 strided convsstrides 5·2·2·2·2·2·2 = 320→ 50 frames/s (20 ms), field 400 = 25 msalso WaveNet-style generatorsspectrogram modelSTFT (n_fft 400, hop 160) → |·|²→ 80 mel bands → log→ 80 × 100 per secondWhisper, Conformer, TTS models,audio classifiers (as an image)compression: 16,000 samples/s → 50 vectors/s (waveform) or 100 frames × 80 bands (log-mel)
fig.From sound to model input. Top: pressure is low-pass filtered below fs/2, then sampled and quantised; 16-bit PCM at 16 kHz is 256 kbit/s, CD stereo is 1411.2 kbit/s. Middle: the data loader resamples (44.1 → 16 kHz is the rational ratio 160/441), scales int16 to [−1, 1), optionally pre-emphasises, and frames into 25 ms windows every 10 ms. Bottom: a waveform model (wav2vec 2.0's 7-layer CNN, total stride 320) or a spectrogram model on 80 log-mel bands.

Quantisation, bit depth and decibels

Each sample is rounded to one of 2b2^b levels. The rounding error behaves like white noise of power Δ2/12\Delta^2/12 for a step Δ\Delta. For a full-scale sine this gives the rule of thumb of about 6 dB per bit:

SQNR≈6.02 b+1.76 dB(b=16: 98.1 dB;  b=24: 146.2 dB)\mathrm{SQNR} \approx 6.02\,b + 1.76\ \text{dB}\qquad(b=16:\ 98.1\ \text{dB};\ \ b=24:\ 146.2\ \text{dB})
bb
bits per sample; 6.02=20log⁡1026.02 = 20\log_{10}2, one bit doubles the number of levels
1.761.76
10log⁡10(3/2)10\log_{10}(3/2): ratio of a full-scale sine's power to the noise power Δ2/12\Delta^2/12, beyond the 22b2^{2b} term

Levels are given in decibels: 20log⁡1020\log_{10} of an amplitude ratio, or 10log⁡1010\log_{10} of a power ratio. In software the reference is full scale (dBFS, where 0 dBFS is the largest sample). In air it is 20 µPa (dB SPL). Perceived loudness is measured in LUFS (ITU-R BS.1770); broadcast normalises to −23 LUFS under EBU R128. A signal at −6 dBFS uses half the range and loses 6 dB of SQNR.

Some systems spend bits non-uniformly. G.711 telephony uses 8-bit μ-law (μ=255\mu=255) at 8 kHz, which gives 64 kbit/s. WaveNet predicts 8-bit μ-law values as a 256-way softmax.

Common formats

Telephony
8 kHz, 8-bit μ-law or A-law (G.711), 64 kbit/s, band 300–3400 Hz
Speech models
16 kHz mono: wav2vec 2.0, HuBERT, Whisper; 16-bit PCM is 256 kbit/s
TTS
22.05 kHz (LJSpeech, HiFi-GAN configs) or 24 kHz
CD audio
44.1 kHz, 16-bit, stereo: 1411.2 kbit/s
Video / studio
48 kHz (and multiples), 24-bit
Float audio
libraries load samples as float32 in [−1, 1): int16 divided by 32768
x(t) = sin(2π·1.1 kHz·t) · fs = 16 kHz → Nyquist fs/2 = 8 kHz · 4-bit+1−100.511.52 ms— tone · ● 33 samples · — 4-bit staircase (16 levels)frequency axis · shaded: 0 … fs/2 · lines at k·fs/204812162024 kHzf < fs/2: 14.55 samples per period, the samples determine the tone uniquelyquantisation: step Δ = 2/2^4 = 0.125 · SQNR measured over 4096 samples = 25.3 dBfull-scale sine rule 6.02·b + 1.76 = 25.8 dB · bit rate fs × b = 64 kbit/s (mono PCM)a real ADC low-pass filters below fs/2 first, so tones above it are removed instead of folded
sample rate fs
fig.A full-scale sine sampled and quantised, with the SQNR measured over 4096 samples. Default (1.1 kHz, fs = 16 kHz, 4-bit): 14.55 samples per period, measured SQNR 25.3 dB against the rule's 25.8 dB. Raise the tone to 15 kHz: the 33 samples in 2 ms fall exactly on a sign-flipped 1 kHz sine (dashed). 9 kHz folds to 7 kHz. At 16 bits the measured SQNR is 98.2 dB against the rule's 98.1 dB. Switch to 8 kHz and set f = 12 kHz: every sample is 0.

Preprocessing in the data loader

Resampling. A model trained at 16 kHz must get 16 kHz input. Rational resampling by L/ML/M upsamples by LL, low-pass filters below the lower Nyquist, and keeps every MM-th sample. Polyphase filters avoid computing the discarded samples. 44.1 → 16 kHz is L/M=160/441L/M = 160/441, since gcd⁡(44100,16000)=100\gcd(44100, 16000) = 100.

Scaling and channels. Stereo is usually averaged to mono. int16 is divided by 32768. Datasets often peak-normalise or loudness-normalise clips, so that level is not a shortcut feature.

Pre-emphasis. Voiced speech loses energy with frequency. A first-order high-pass boosts the highs before analysis:

y[n]=x[n]−α x[n−1],∣H(f)∣=∣1−α e−j2πf/fs∣y[n] = x[n] - \alpha\,x[n-1],\qquad |H(f)| = \big|1-\alpha\,e^{-j2\pi f/f_s}\big|
α\alpha
typically 0.95–0.97; at α=0.97\alpha=0.97 and 16 kHz the gain is −30.5 dB at DC, −8.3 dB at 1 kHz, +2.9 dB at 4 kHz and +5.9 dB at 8 kHz

Pre-emphasis belongs to the classic MFCC front end and to some TTS recipes. Whisper and most learned front ends skip it.

Framing. Speech is roughly stationary over 10–30 ms, so analysis uses overlapping frames: 25 ms windows every 10 ms (400 and 160 samples at 16 kHz), or 100 frames per second. Each frame is windowed and transformed, see Short-time Fourier transform (STFT).

Worked example: a 10 s music clip to model inputs

StageComputationResult
Raw PCM441,000 samples × 2 channels × 2 bytes1,764,000 bytes
Mono, resampled to 16 kHz10 × 16,000 samples, float32160,000 samples = 640,000 bytes
Frames, no padding1 + ⌊(160,000 − 400)/160⌋998 frames of 400
Log-mel (centred STFT)(1 + 160,000/160) frames × 80 bands80 × 1001 = 80,080 values
wav2vec 2.0 CNN7 convs, kernels 10,3,3,3,3,2,2, strides 5,2,2,2,2,2,2499 frames (stride 320 = 20 ms, field 400 = 25 ms)
Sizes at each stage for a 10 s, 44.1 kHz, 16-bit stereo clip
python
1import numpy as np
2
3fs = 16_000
4t = np.arange(fs) / fs # 1 s
5x = 0.5 * np.sin(2 * np.pi * 220 * t) + 0.01 * np.random.default_rng(0).standard_normal(fs)
6
7# Quantise to int16 PCM and back, as a WAV file would
8pcm = np.clip(np.round(x * 32768), -32768, 32767).astype(np.int16)
9y = pcm.astype(np.float32) / 32768.0
10print("int16 round-trip SNR:", round(10 * np.log10(np.sum(x**2) / np.sum((y - x)**2)), 1), "dB")
11# 92.1 dB: the 98.1 dB of full scale minus 6 dB, since the signal uses half the range
12
13# Aliasing: a 15 kHz tone sampled at 16 kHz gives the same samples as a 1 kHz tone (sign flipped)
14n = np.arange(64)
15print(np.allclose(np.sin(2*np.pi*15_000*n/fs), -np.sin(2*np.pi*1_000*n/fs))) # True
16
17# Peak level in dBFS and pre-emphasis
18print("peak:", round(20 * np.log10(np.abs(y).max()), 2), "dBFS") # -5.5 dBFS
19y_pre = np.append(y[0], y[1:] - 0.97 * y[:-1])
20
21# Framing: 25 ms windows every 10 ms -> (frames, 400)
22win, hop = int(0.025 * fs), int(0.010 * fs)
23frames = np.lib.stride_tricks.sliding_window_view(y_pre, win)[::hop]
24print("frames:", frames.shape) # (98, 400) = 1 + (16000 - 400) // 160
PCM round trip, aliasing, dBFS, pre-emphasis and framing in numpy (runs as is; for resampling use torchaudio.functional.resample or scipy.signal.resample_poly)

Waveform or spectrogram input?

A spectrogram model starts from a fixed transform: STFT magnitude, mel filterbank, log (see Mel spectrograms). The input is a compact 80 × 100 image per second that discards phase. Whisper, Conformer recognisers and most audio classifiers work this way. A waveform model learns its front end. wav2vec 2.0 and HuBERT use a strided CNN that reduces 16,000 samples per second to 50 vectors. Generators such as WaveNet and HiFi-GAN must output waveforms, so they operate at the sample rate.

Use when

  • +Spectrogram input: compact, matches perception, cheap to augment (SpecAugment); a sound default for recognition and classification.
  • +Waveform input: nothing discarded (phase, fine timing); the front end is learned end to end; required for generation.

Avoid when

  • −Spectrogram input: phase is lost, so generating audio needs a vocoder; front-end settings must match between training and inference.
  • −Waveform input: 16k–48k steps per second; costly at the sample rate and sensitive to the level and DC offset.

Check yourself

A 15 kHz tone reaches a 16 kHz ADC whose anti-aliasing filter is missing. What do the samples contain?

Check yourself

Going from 16-bit to 24-bit samples raises the quantisation SNR of a full-scale sine by about how much?

Remember

  • ✓Sampling at fsf_s captures everything below fs/2f_s/2; anything above aliases to ∣f−fs round(f/fs)∣|f - f_s\,\mathrm{round}(f/f_s)| and must be filtered out first.
  • ✓16 kHz for speech models, 22.05/24 kHz for TTS, 44.1/48 kHz for music; 8 kHz telephony keeps only 300–3400 Hz.
  • ✓Quantisation SQNR ≈ 6.02·b + 1.76 dB: 98 dB at 16 bits; level in dBFS, loudness in LUFS.
  • ✓Loader chain: resample (polyphase, e.g. 160/441), mono, int16 / 32768, optional pre-emphasis (α ≈ 0.97), 25 ms / 10 ms frames.
  • ✓Waveform models learn the front end (wav2vec 2.0: stride 320, 50 frames/s); spectrogram models start from log-mel (80 × 100 per second).
  • ✓Match sample rate and front-end settings between training and inference; a mismatch fails silently.
← back to Deep Learning roadmap