Definition
Audio signal processing for deep learning is the fixed chain that turns a sound wave into tensors: sampling and quantisation in the converter, then resampling, scaling, optional pre-emphasis and framing in the data loader. Its theory rests on the sampling theorem (Nyquist, 1928; Shannon, "Communication in the Presence of Noise", 1949): a signal with no energy at or above is fully determined by its samples taken times per second.
Sound is a small, fast variation of air pressure around the atmospheric mean. It travels at about 343 m/s at 20 °C, and humans hear roughly 20 Hz to 20 kHz. A microphone turns pressure into a voltage. An analogue-to-digital converter (ADC) turns the voltage into a sequence of integers. Every choice made here (sample rate, bit depth, filters) limits what any later model can learn.
The model never sees "sound". It sees a 1-D array of about 16,000 numbers per second of speech, or 44,100 to 48,000 for music. It either learns directly from that array, or from a time–frequency image computed from it.
Sampling and aliasing
Sampling keeps the values . The Nyquist frequency is . Any tone above it produces exactly the same samples as a tone inside , so after sampling the two cannot be told apart. This is aliasing:
- frequency of the analogue tone
- sample rate; is the Nyquist frequency
- the frequency the samples actually represent; e.g. 15 kHz at kHz → 1 kHz
Aliasing cannot be undone after sampling. Converters therefore apply an anti-aliasing low-pass filter before the ADC, and every resampler applies one before it reduces the rate. Telephony runs at 8 kHz with a 300–3400 Hz band. Speech recognition uses 16 kHz, which keeps everything below 8 kHz, where almost all speech information lies. TTS models often use 22.05 or 24 kHz. Music uses 44.1 kHz (CD) or 48 kHz (video and studio), just above twice the 20 kHz hearing limit.
Quantisation, bit depth and decibels
Each sample is rounded to one of levels. The rounding error behaves like white noise of power for a step . For a full-scale sine this gives the rule of thumb of about 6 dB per bit:
- bits per sample; , one bit doubles the number of levels
- : ratio of a full-scale sine's power to the noise power , beyond the term
Levels are given in decibels: of an amplitude ratio, or of a power ratio. In software the reference is full scale (dBFS, where 0 dBFS is the largest sample). In air it is 20 µPa (dB SPL). Perceived loudness is measured in LUFS (ITU-R BS.1770); broadcast normalises to −23 LUFS under EBU R128. A signal at −6 dBFS uses half the range and loses 6 dB of SQNR.
Some systems spend bits non-uniformly. G.711 telephony uses 8-bit μ-law () at 8 kHz, which gives 64 kbit/s. WaveNet predicts 8-bit μ-law values as a 256-way softmax.
Common formats
- Telephony
- 8 kHz, 8-bit μ-law or A-law (G.711), 64 kbit/s, band 300–3400 Hz
- Speech models
- 16 kHz mono: wav2vec 2.0, HuBERT, Whisper; 16-bit PCM is 256 kbit/s
- TTS
- 22.05 kHz (LJSpeech, HiFi-GAN configs) or 24 kHz
- CD audio
- 44.1 kHz, 16-bit, stereo: 1411.2 kbit/s
- Video / studio
- 48 kHz (and multiples), 24-bit
- Float audio
- libraries load samples as float32 in [−1, 1): int16 divided by 32768
Preprocessing in the data loader
Resampling. A model trained at 16 kHz must get 16 kHz input. Rational resampling by upsamples by , low-pass filters below the lower Nyquist, and keeps every -th sample. Polyphase filters avoid computing the discarded samples. 44.1 → 16 kHz is , since .
Scaling and channels. Stereo is usually averaged to mono. int16 is divided by 32768. Datasets often peak-normalise or loudness-normalise clips, so that level is not a shortcut feature.
Pre-emphasis. Voiced speech loses energy with frequency. A first-order high-pass boosts the highs before analysis:
- typically 0.95–0.97; at and 16 kHz the gain is −30.5 dB at DC, −8.3 dB at 1 kHz, +2.9 dB at 4 kHz and +5.9 dB at 8 kHz
Pre-emphasis belongs to the classic MFCC front end and to some TTS recipes. Whisper and most learned front ends skip it.
Framing. Speech is roughly stationary over 10–30 ms, so analysis uses overlapping frames: 25 ms windows every 10 ms (400 and 160 samples at 16 kHz), or 100 frames per second. Each frame is windowed and transformed, see Short-time Fourier transform (STFT).
Worked example: a 10 s music clip to model inputs
| Stage | Computation | Result |
|---|---|---|
| Raw PCM | 441,000 samples × 2 channels × 2 bytes | 1,764,000 bytes |
| Mono, resampled to 16 kHz | 10 × 16,000 samples, float32 | 160,000 samples = 640,000 bytes |
| Frames, no padding | 1 + ⌊(160,000 − 400)/160⌋ | 998 frames of 400 |
| Log-mel (centred STFT) | (1 + 160,000/160) frames × 80 bands | 80 × 1001 = 80,080 values |
| wav2vec 2.0 CNN | 7 convs, kernels 10,3,3,3,3,2,2, strides 5,2,2,2,2,2,2 | 499 frames (stride 320 = 20 ms, field 400 = 25 ms) |
1import numpy as np23fs = 16_0004t = np.arange(fs) / fs # 1 s5x = 0.5 * np.sin(2 * np.pi * 220 * t) + 0.01 * np.random.default_rng(0).standard_normal(fs)67# Quantise to int16 PCM and back, as a WAV file would8pcm = np.clip(np.round(x * 32768), -32768, 32767).astype(np.int16)9y = pcm.astype(np.float32) / 32768.010print("int16 round-trip SNR:", round(10 * np.log10(np.sum(x**2) / np.sum((y - x)**2)), 1), "dB")11# 92.1 dB: the 98.1 dB of full scale minus 6 dB, since the signal uses half the range1213# Aliasing: a 15 kHz tone sampled at 16 kHz gives the same samples as a 1 kHz tone (sign flipped)14n = np.arange(64)15print(np.allclose(np.sin(2*np.pi*15_000*n/fs), -np.sin(2*np.pi*1_000*n/fs))) # True1617# Peak level in dBFS and pre-emphasis18print("peak:", round(20 * np.log10(np.abs(y).max()), 2), "dBFS") # -5.5 dBFS19y_pre = np.append(y[0], y[1:] - 0.97 * y[:-1])2021# Framing: 25 ms windows every 10 ms -> (frames, 400)22win, hop = int(0.025 * fs), int(0.010 * fs)23frames = np.lib.stride_tricks.sliding_window_view(y_pre, win)[::hop]24print("frames:", frames.shape) # (98, 400) = 1 + (16000 - 400) // 160Waveform or spectrogram input?
A spectrogram model starts from a fixed transform: STFT magnitude, mel filterbank, log (see Mel spectrograms). The input is a compact 80 × 100 image per second that discards phase. Whisper, Conformer recognisers and most audio classifiers work this way. A waveform model learns its front end. wav2vec 2.0 and HuBERT use a strided CNN that reduces 16,000 samples per second to 50 vectors. Generators such as WaveNet and HiFi-GAN must output waveforms, so they operate at the sample rate.
Use when
- +Spectrogram input: compact, matches perception, cheap to augment (SpecAugment); a sound default for recognition and classification.
- +Waveform input: nothing discarded (phase, fine timing); the front end is learned end to end; required for generation.
Avoid when
- −Spectrogram input: phase is lost, so generating audio needs a vocoder; front-end settings must match between training and inference.
- −Waveform input: 16k–48k steps per second; costly at the sample rate and sensitive to the level and DC offset.
Check yourself
A 15 kHz tone reaches a 16 kHz ADC whose anti-aliasing filter is missing. What do the samples contain?
Check yourself
Going from 16-bit to 24-bit samples raises the quantisation SNR of a full-scale sine by about how much?
Remember
- ✓Sampling at captures everything below ; anything above aliases to and must be filtered out first.
- ✓16 kHz for speech models, 22.05/24 kHz for TTS, 44.1/48 kHz for music; 8 kHz telephony keeps only 300–3400 Hz.
- ✓Quantisation SQNR ≈ 6.02·b + 1.76 dB: 98 dB at 16 bits; level in dBFS, loudness in LUFS.
- ✓Loader chain: resample (polyphase, e.g. 160/441), mono, int16 / 32768, optional pre-emphasis (α ≈ 0.97), 25 ms / 10 ms frames.
- ✓Waveform models learn the front end (wav2vec 2.0: stride 320, 50 frames/s); spectrogram models start from log-mel (80 × 100 per second).
- ✓Match sample rate and front-end settings between training and inference; a mismatch fails silently.