Speech enhancement

Speech enhancement recovers clean speech from noisy, reverberant audio by predicting a time–frequency mask (IBM, IRM, cIRM) or the waveform itself (Conv-TasNet, Demucs); trained on SI-SDR, judged also by PESQ and STOI.

8 min read898 wordscore0 viewsupdated 2026-10-15

Definition

Speech enhancement estimates the clean speech ss from a degraded recording y=s+ny=s+n, or y=h∗s+ny=h*s+n when the room adds reverberation hh. It covers denoising and dereverberation, from one microphone (monaural) or an array. Classical methods are spectral subtraction (Boll, 1979), the Wiener filter and the MMSE short-time spectral amplitude estimator (Ephraim and Malah, 1984). Supervised deep learning reframed the task as predicting a mask (Wang, Narayanan and Wang, "On Training Targets for Supervised Speech Separation", 2014) or the waveform directly. Examples of the latter are Conv-TasNet (Luo and Mesgarani, 2019) and the Demucs denoiser (Défossez, Synnaeve and Adi, Interspeech 2020).

The input is a noisy waveform and the output is a waveform of the same length. The noise can be stationary (fan hum) or anything but (babble, keyboard clicks, a dog). It can be louder than the speech, and it overlaps the speech in time and in frequency. Removing it must not remove speech. In calls and hearing aids, the model must also run causally with a few tens of milliseconds of latency. Enhancement often sits in front of Speech-to-text or Speaker diarization.

Classical methods estimate the noise spectrum during pauses and subtract it. Spectral subtraction sets ∣S^∣2=max⁡(∣Y∣2−λ^N, β∣Y∣2)|\hat S|^2=\max(|Y|^2-\hat\lambda_N,\ \beta|Y|^2). Bins where the noise momentarily exceeds its average survive as isolated tones, the "musical noise" artefact. These methods assume the noise is stationary, which real noise rarely is.

Masking in the STFT domain

Take the Short-time Fourier transform (STFT). Then Y(t,f)=S(t,f)+N(t,f)Y(t,f)=S(t,f)+N(t,f) in every bin. Speech energy is sparse: at any moment it sits in a few harmonics and formants, so many bins are dominated by either speech or noise. A network (LSTM, convolutional recurrent network or U-Net) reads ∣Y∣|Y| and outputs a mask M(t,f)M(t,f). The estimate is S^=M⊙Y\hat S=M\odot Y, inverted by overlap-add. The training target is an ideal mask computed from the known SS and NN of a synthetic mixture:

IBM=1 ⁣[∣S∣2∣N∣2>10LC/10],IRM=(∣S∣2∣S∣2+∣N∣2)β,GWiener=ξ1+ξ\mathrm{IBM}=\mathbb 1\!\Big[\tfrac{|S|^2}{|N|^2}>10^{\mathrm{LC}/10}\Big],\qquad \mathrm{IRM}=\Big(\frac{|S|^2}{|S|^2+|N|^2}\Big)^{\beta},\qquad G_{\text{Wiener}}=\frac{\xi}{1+\xi}
LC\mathrm{LC}
local criterion in dB (often 0 dB): keep a bin if speech dominates it
β\beta
compression of the ratio mask; β=12\beta=\tfrac12 is the usual IRM
ξ\xi
a priori SNR λS/λN\lambda_S/\lambda_N of the bin; with oracle powers the Wiener gain is the IRM with β=1\beta=1
noisy y = s + n (+ reverberation) → estimate ŝ of the clean speech sA · time–frequency maskingnoisy ySTFT|Y| and ∠YT × F binsmask netLSTM / CRN /U-Net on |Y|M ⊙ YIBM / IRMor cIRMiSTFToverlap-addnoisy ∠Yestimate ŝ|Y|⊙mask M ∈ [0, 1]=|Ŝ|real mask: phasestays noisy; cIRMfixes re and imB · waveform model (Conv-TasNet, 8 kHz; Demucs denoiser is a waveform U-Net)encoder1-D conv, N = 512L = 16, stride 8separator (TCN)dilated 1-D convblocks → masksdecodertransposed convback to sampleslearned basis (2 ms filters) replaces the STFT; no phase to estimatetraining loss−SI-SDR on the waveform, orL1 + multi-resolution STFTevaluationSI-SDR (dB) · PESQ −0.5…4.5STOI 0…1 · DNSMOS (no reference)
fig.Two families. A: time–frequency masking. STFT, a network predicts a mask from |Y|, the mask multiplies the noisy spectrogram, and the inverse STFT reuses the noisy phase ∠Y. A complex mask (cIRM) corrects real and imaginary parts instead. B: a waveform model in Conv-TasNet's layout. A learned 1-D conv encoder (512 filters of 16 samples, stride 8, i.e. 2 ms at 8 kHz) replaces the STFT. A dilated temporal convolutional network predicts masks on that representation, and a transposed-conv decoder returns samples. Bottom: training losses and evaluation metrics.

A real-valued mask scales only the magnitude, so the output keeps the noisy phase. At high SNR that hardly matters. At 0 dB and below, the phase is badly wrong in many speech bins, and it caps the quality. Two fixes are common. The complex ratio mask (Williamson, Wang and Wang, 2016) predicts M=S/YM=S/Y with real and imaginary parts, which corrects phase as well. Its components are Mr=(YrSr+YiSi)/∣Y∣2M_r=(Y_rS_r+Y_iS_i)/|Y|^2 and Mi=(YrSi−YiSr)/∣Y∣2M_i=(Y_rS_i-Y_iS_r)/|Y|^2, compressed with a tanh for training. The other fix is to leave the STFT altogether.

Waveform models

Conv-TasNet ("Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation", IEEE/ACM TASLP 2019) was built for separating speakers. It replaces the STFT with a learned encoder: a 1-D convolution with N=512N=512 filters of L=16L=16 samples and stride 8. A temporal convolutional network then predicts one mask per source on that representation. It stacks 8 dilated blocks (dilations 1 to 128), repeated 3 times. A transposed convolution decodes back to samples. There is no phase to estimate, because the basis is learned and real-valued. The Demucs denoiser is a causal waveform U-Net with an LSTM bottleneck. It is trained with an L1 waveform loss plus a multi-resolution STFT loss and runs in real time on a CPU.

starget=⟨s^,s⟩∥s∥2 s,SI-SDR=10log⁡10∥starget∥2∥s^−starget∥2s_{\text{target}}=\frac{\langle \hat s,s\rangle}{\lVert s\rVert^2}\,s,\qquad \mathrm{SI\text{-}SDR}=10\log_{10}\frac{\lVert s_{\text{target}}\rVert^2}{\lVert \hat s-s_{\text{target}}\rVert^2}
stargets_{\text{target}}
projection of the estimate onto the clean signal; rescaling s^\hat s does not change the score
s^−starget\hat s-s_{\text{target}}
everything else: residual noise, artefacts and distortion
ΔSI-SDR\Delta\text{SI-SDR}
improvement over the noisy input, the usual headline number (Le Roux, Wisdom, Erdogan and Hershey, ICASSP 2019)

Waveform models are usually trained on −-SI-SDR directly. For separation, the loss is taken over the best speaker permutation, as in Speaker diarization's EEND. Metrics that track human judgement are reported alongside. PESQ (ITU-T P.862, wideband P.862.2) compares the output with the clean reference and predicts quality on a −0.5 to 4.5 scale. STOI (Taal et al., 2011) predicts intelligibility from 0 to 1. DNSMOS predicts MOS without any reference, so it works on real recordings.

STFT: 256-pt sqrt-Hann, hop 128, 8 kHz · 33 × 129 · SNR 0 dB, whiteŜ = M ⊙ Y, resynthesised with the noisy phase ∠Y by overlap-addnoisy |Y| (dB)00.51 smask M: IRM (β = ½)00.51 senhanced |M·Y| (dB)00.51 s0–4 kHz ↑mean M = 0.172SI-SDR vs the clean signal (dB), all four masks at this setting-100102030noisy input0.09IBM11.47IRM (β = ½)11.19Wiener12.30spectral subtraction3.71IRM (β = ½): 0.09 → 11.19 dB, ΔSI-SDR = +11.09 dBIBM, IRM, Wiener are oracle (built from clean S and noise N); spec. sub. knows only the noise PSD
mask
noisephase
fig.A synthetic two-syllable voiced signal (f0 140 → 182 Hz, three formants) mixed with seeded noise, 8 kHz, 256-point STFT, hop 128. Default (white noise, 0 dB, IRM, noisy phase): SI-SDR goes from 0.09 dB to 11.19 dB (+11.09). The IBM keeps only 9.3% of the bins and reaches 11.47 dB, Wiener 12.30 dB, and spectral subtraction, which knows only the average noise power, 3.71 dB. Switching to the oracle clean phase lifts the same IRM to 17.68 dB, a 6.50 dB gap that is the cost of the noisy phase. At −5 dB, spectral subtraction gives −1.39 dB while Wiener still gives 9.09 dB.
Local SNRIBM (LC 0 dB)IRM (β = ½)Wiener (β = 1)IRM / Wiener attenuation
+10 dB (ratio 10)1√(10/11) = 0.95310/11 = 0.909−0.41 dB / −0.83 dB
0 dB (ratio 1)0√(1/2) = 0.7071/2 = 0.500−3.01 dB / −6.02 dB
−10 dB (ratio 0.1)0√(1/11) = 0.3021/11 = 0.091−10.41 dB / −20.83 dB
Worked example: what each ideal mask does to one bin, by local SNR |S|²/|N|²

The IBM is all or nothing, so speech in noise-dominated bins is lost and the output can sound choppy. The ratio masks attenuate smoothly. With the noise power set to its true value and no cross term, ∣Y∣2=∣S∣2+∣N∣2|Y|^2=|S|^2+|N|^2, the spectral-subtraction gain 1−∣N∣2/∣Y∣2\sqrt{1-|N|^2/|Y|^2} is exactly the IRM. Spectral subtraction falls short only because it must use an average noise power instead of the true one, bin by bin.

Reference points

Conv-TasNet
WSJ0-2mix (8 kHz, two speakers): SI-SNR improvement 15.3 dB with 5.1M parameters, above ideal magnitude masks
IRM
β = 0.5 (Wang, Narayanan and Wang, 2014); the cIRM extends it to complex values (Williamson et al., 2016)
PESQ / STOI
PESQ: ITU-T P.862 (2001), wideband P.862.2, scale −0.5 to 4.5; STOI: 0 to 1, intelligibility
DNS Challenge
Microsoft's Deep Noise Suppression challenge, first at Interspeech 2020: open clean speech and noise sets, a mixing script, ITU-T P.808 crowdsourced MOS
VoiceBank+DEMAND
Valentini-Botinhao et al. (2016): 28 training speakers, 824 test utterances; the standard small denoising benchmark
Dereverberation
WPE (Nakatani et al., 2010): multi-frame linear prediction in the STFT domain; neural models target the direct path plus early reflections

Building supervised training pairs

  1. 1Draw a clean utterance ss and a noise clip nn from separate corpora; optionally convolve ss with a room impulse response hh.
  2. 2Pick a random SNR and scale the noise: n←n∥s∥2/(∥n∥2 10SNR/10)n\leftarrow n\sqrt{\lVert s\rVert^2/(\lVert n\rVert^2\,10^{\mathrm{SNR}/10})}; mix y=s+ny=s+n.
  3. 3Mask models: compute S,N,YS,N,Y by STFT and the target mask (IRM, cIRM) per bin. Waveform models: the target is ss itself.
  4. 4Train on random crops of a few seconds; the mixtures are generated on the fly, so the model rarely sees the same pair twice.

Where it's used

Calls and conferencing

causal noise suppression on each device, the setting of the DNS Challenge

Hearing aids

low-latency enhancement where intelligibility (STOI) matters more than polish

ASR front end

denoising before Speech-to-text, though enhancement artefacts can also raise WER

Media cleanup

offline removal of hum, wind and reverberation from recordings, without latency limits

python
1import numpy as np, soundfile as sf
2from pesq import pesq
3from pystoi import stoi
4
5def stft(x, n_fft=512, hop=256):
6 w = np.sqrt(np.hanning(n_fft + 1)[:-1]) # periodic sqrt-Hann: perfect reconstruction at 50 %
7 x = np.pad(x, (hop, hop + n_fft))
8 return np.fft.rfft(np.lib.stride_tricks.sliding_window_view(x, n_fft)[::hop] * w, axis=-1)
9
10def istft(X, length, n_fft=512, hop=256):
11 w = np.sqrt(np.hanning(n_fft + 1)[:-1])
12 frames = np.fft.irfft(X, n_fft, axis=-1) * w
13 y = np.zeros(hop * (len(frames) - 1) + n_fft)
14 for k, f in enumerate(frames):
15 y[k * hop : k * hop + n_fft] += f
16 return y[hop : hop + length]
17
18def si_sdr(est, ref):
19 t = (est @ ref) / (ref @ ref) * ref # project onto the clean signal
20 return 10 * np.log10((t @ t) / ((est - t) @ (est - t)))
21
22s, fs = sf.read("clean.wav") # 16 kHz mono
23n, _ = sf.read("noise.wav")
24n = n[: len(s)] * np.sqrt((s @ s) / (n @ n)) # scale noise for 0 dB SNR
25y = s + n
26S, N, Y = stft(s), stft(n), stft(y)
27irm = np.sqrt(np.abs(S) ** 2 / (np.abs(S) ** 2 + np.abs(N) ** 2 + 1e-12))
28s_hat = istft(irm * Y, len(y)) # the oracle target a network would learn
29
30print(f"SI-SDR {si_sdr(y, s):.2f} -> {si_sdr(s_hat, s):.2f} dB")
31print("PESQ-WB", pesq(fs, s, s_hat, "wb"), " STOI", stoi(s, s_hat, fs))
Ideal ratio mask with a perfect-reconstruction STFT, plus SI-SDR, PESQ and STOI (the numpy part was run on synthetic signals; the soundfile, pesq and pystoi calls were not)

Check yourself

A bin has |S|² = |N|² (0 dB local SNR). What do the IBM with LC = 0 dB, the IRM with β = ½ and the Wiener gain output?

Check yourself

Why does SI-SDR first project the estimate onto the clean signal?

Remember

  • ✓Model: y=s+ny=s+n (or h∗s+nh*s+n); recover ss, causally for real-time use.
  • ✓Masks: IBM (binary), IRM =(∣S∣2/(∣S∣2+∣N∣2))1/2=(|S|^2/(|S|^2+|N|^2))^{1/2}, Wiener =ξ/(1+ξ)=\xi/(1+\xi), cIRM (complex, fixes phase).
  • ✓Magnitude masks keep the noisy phase; in the viz that costs 6.50 dB of SI-SDR at 0 dB SNR.
  • ✓Waveform models: Conv-TasNet (learned 512 × 16-sample encoder, dilated TCN), Demucs (causal waveform U-Net).
  • ✓Train on −SI-SDR (or L1 + multi-resolution STFT); report ΔSI-SDR, PESQ, STOI, DNSMOS.
← back to Deep Learning roadmap