Definition
Speech enhancement estimates the clean speech from a degraded recording , or when the room adds reverberation . It covers denoising and dereverberation, from one microphone (monaural) or an array. Classical methods are spectral subtraction (Boll, 1979), the Wiener filter and the MMSE short-time spectral amplitude estimator (Ephraim and Malah, 1984). Supervised deep learning reframed the task as predicting a mask (Wang, Narayanan and Wang, "On Training Targets for Supervised Speech Separation", 2014) or the waveform directly. Examples of the latter are Conv-TasNet (Luo and Mesgarani, 2019) and the Demucs denoiser (Défossez, Synnaeve and Adi, Interspeech 2020).
The input is a noisy waveform and the output is a waveform of the same length. The noise can be stationary (fan hum) or anything but (babble, keyboard clicks, a dog). It can be louder than the speech, and it overlaps the speech in time and in frequency. Removing it must not remove speech. In calls and hearing aids, the model must also run causally with a few tens of milliseconds of latency. Enhancement often sits in front of Speech-to-text or Speaker diarization.
Classical methods estimate the noise spectrum during pauses and subtract it. Spectral subtraction sets . Bins where the noise momentarily exceeds its average survive as isolated tones, the "musical noise" artefact. These methods assume the noise is stationary, which real noise rarely is.
Masking in the STFT domain
Take the Short-time Fourier transform (STFT). Then in every bin. Speech energy is sparse: at any moment it sits in a few harmonics and formants, so many bins are dominated by either speech or noise. A network (LSTM, convolutional recurrent network or U-Net) reads and outputs a mask . The estimate is , inverted by overlap-add. The training target is an ideal mask computed from the known and of a synthetic mixture:
- local criterion in dB (often 0 dB): keep a bin if speech dominates it
- compression of the ratio mask; is the usual IRM
- a priori SNR of the bin; with oracle powers the Wiener gain is the IRM with
A real-valued mask scales only the magnitude, so the output keeps the noisy phase. At high SNR that hardly matters. At 0 dB and below, the phase is badly wrong in many speech bins, and it caps the quality. Two fixes are common. The complex ratio mask (Williamson, Wang and Wang, 2016) predicts with real and imaginary parts, which corrects phase as well. Its components are and , compressed with a tanh for training. The other fix is to leave the STFT altogether.
Waveform models
Conv-TasNet ("Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation", IEEE/ACM TASLP 2019) was built for separating speakers. It replaces the STFT with a learned encoder: a 1-D convolution with filters of samples and stride 8. A temporal convolutional network then predicts one mask per source on that representation. It stacks 8 dilated blocks (dilations 1 to 128), repeated 3 times. A transposed convolution decodes back to samples. There is no phase to estimate, because the basis is learned and real-valued. The Demucs denoiser is a causal waveform U-Net with an LSTM bottleneck. It is trained with an L1 waveform loss plus a multi-resolution STFT loss and runs in real time on a CPU.
- projection of the estimate onto the clean signal; rescaling does not change the score
- everything else: residual noise, artefacts and distortion
- improvement over the noisy input, the usual headline number (Le Roux, Wisdom, Erdogan and Hershey, ICASSP 2019)
Waveform models are usually trained on SI-SDR directly. For separation, the loss is taken over the best speaker permutation, as in Speaker diarization's EEND. Metrics that track human judgement are reported alongside. PESQ (ITU-T P.862, wideband P.862.2) compares the output with the clean reference and predicts quality on a −0.5 to 4.5 scale. STOI (Taal et al., 2011) predicts intelligibility from 0 to 1. DNSMOS predicts MOS without any reference, so it works on real recordings.
| Local SNR | IBM (LC 0 dB) | IRM (β = ½) | Wiener (β = 1) | IRM / Wiener attenuation |
|---|---|---|---|---|
| +10 dB (ratio 10) | 1 | √(10/11) = 0.953 | 10/11 = 0.909 | −0.41 dB / −0.83 dB |
| 0 dB (ratio 1) | 0 | √(1/2) = 0.707 | 1/2 = 0.500 | −3.01 dB / −6.02 dB |
| −10 dB (ratio 0.1) | 0 | √(1/11) = 0.302 | 1/11 = 0.091 | −10.41 dB / −20.83 dB |
The IBM is all or nothing, so speech in noise-dominated bins is lost and the output can sound choppy. The ratio masks attenuate smoothly. With the noise power set to its true value and no cross term, , the spectral-subtraction gain is exactly the IRM. Spectral subtraction falls short only because it must use an average noise power instead of the true one, bin by bin.
Reference points
- Conv-TasNet
- WSJ0-2mix (8 kHz, two speakers): SI-SNR improvement 15.3 dB with 5.1M parameters, above ideal magnitude masks
- IRM
- β = 0.5 (Wang, Narayanan and Wang, 2014); the cIRM extends it to complex values (Williamson et al., 2016)
- PESQ / STOI
- PESQ: ITU-T P.862 (2001), wideband P.862.2, scale −0.5 to 4.5; STOI: 0 to 1, intelligibility
- DNS Challenge
- Microsoft's Deep Noise Suppression challenge, first at Interspeech 2020: open clean speech and noise sets, a mixing script, ITU-T P.808 crowdsourced MOS
- VoiceBank+DEMAND
- Valentini-Botinhao et al. (2016): 28 training speakers, 824 test utterances; the standard small denoising benchmark
- Dereverberation
- WPE (Nakatani et al., 2010): multi-frame linear prediction in the STFT domain; neural models target the direct path plus early reflections
Building supervised training pairs
- 1Draw a clean utterance and a noise clip from separate corpora; optionally convolve with a room impulse response .
- 2Pick a random SNR and scale the noise: ; mix .
- 3Mask models: compute by STFT and the target mask (IRM, cIRM) per bin. Waveform models: the target is itself.
- 4Train on random crops of a few seconds; the mixtures are generated on the fly, so the model rarely sees the same pair twice.
Where it's used
Calls and conferencing
causal noise suppression on each device, the setting of the DNS Challenge
Hearing aids
low-latency enhancement where intelligibility (STOI) matters more than polish
ASR front end
denoising before Speech-to-text, though enhancement artefacts can also raise WER
Media cleanup
offline removal of hum, wind and reverberation from recordings, without latency limits
1import numpy as np, soundfile as sf2from pesq import pesq3from pystoi import stoi45def stft(x, n_fft=512, hop=256):6 w = np.sqrt(np.hanning(n_fft + 1)[:-1]) # periodic sqrt-Hann: perfect reconstruction at 50 %7 x = np.pad(x, (hop, hop + n_fft))8 return np.fft.rfft(np.lib.stride_tricks.sliding_window_view(x, n_fft)[::hop] * w, axis=-1)910def istft(X, length, n_fft=512, hop=256):11 w = np.sqrt(np.hanning(n_fft + 1)[:-1])12 frames = np.fft.irfft(X, n_fft, axis=-1) * w13 y = np.zeros(hop * (len(frames) - 1) + n_fft)14 for k, f in enumerate(frames):15 y[k * hop : k * hop + n_fft] += f16 return y[hop : hop + length]1718def si_sdr(est, ref):19 t = (est @ ref) / (ref @ ref) * ref # project onto the clean signal20 return 10 * np.log10((t @ t) / ((est - t) @ (est - t)))2122s, fs = sf.read("clean.wav") # 16 kHz mono23n, _ = sf.read("noise.wav")24n = n[: len(s)] * np.sqrt((s @ s) / (n @ n)) # scale noise for 0 dB SNR25y = s + n26S, N, Y = stft(s), stft(n), stft(y)27irm = np.sqrt(np.abs(S) ** 2 / (np.abs(S) ** 2 + np.abs(N) ** 2 + 1e-12))28s_hat = istft(irm * Y, len(y)) # the oracle target a network would learn2930print(f"SI-SDR {si_sdr(y, s):.2f} -> {si_sdr(s_hat, s):.2f} dB")31print("PESQ-WB", pesq(fs, s, s_hat, "wb"), " STOI", stoi(s, s_hat, fs))Check yourself
A bin has |S|² = |N|² (0 dB local SNR). What do the IBM with LC = 0 dB, the IRM with β = ½ and the Wiener gain output?
Check yourself
Why does SI-SDR first project the estimate onto the clean signal?
Remember
- ✓Model: (or ); recover , causally for real-time use.
- ✓Masks: IBM (binary), IRM , Wiener , cIRM (complex, fixes phase).
- ✓Magnitude masks keep the noisy phase; in the viz that costs 6.50 dB of SI-SDR at 0 dB SNR.
- ✓Waveform models: Conv-TasNet (learned 512 × 16-sample encoder, dilated TCN), Demucs (causal waveform U-Net).
- ✓Train on −SI-SDR (or L1 + multi-resolution STFT); report ΔSI-SDR, PESQ, STOI, DNSMOS.