# Speech enhancement

> Speech enhancement recovers clean speech from noisy, reverberant audio by predicting a time–frequency mask (IBM, IRM, cIRM) or the waveform itself (Conv-TasNet, Demucs); trained on SI-SDR, judged also by PESQ and STOI.

Source: https://www.mlfoundry.in/learn/speech-enhancement  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Speech enhancement:** estimates the clean speech $s$ from a degraded recording $y=s+n$, or $y=h*s+n$ when the room adds reverberation $h$. It covers denoising and dereverberation, from one microphone (monaural) or an array. Classical methods are spectral subtraction (Boll, 1979), the Wiener filter and the MMSE short-time spectral amplitude estimator (Ephraim and Malah, 1984). Supervised deep learning reframed the task as predicting a **mask** (Wang, Narayanan and Wang, "On Training Targets for Supervised Speech Separation", 2014) or the waveform directly. Examples of the latter are Conv-TasNet (Luo and Mesgarani, 2019) and the Demucs denoiser (Défossez, Synnaeve and Adi, Interspeech 2020).

The input is a noisy waveform and the output is a waveform of the same length. The noise can be stationary (fan hum) or anything but (babble, keyboard clicks, a dog). It can be louder than the speech, and it overlaps the speech in time and in frequency. Removing it must not remove speech. In calls and hearing aids, the model must also run causally with a few tens of milliseconds of latency. Enhancement often sits in front of **Speech-to-text** or **Speaker diarization**.

Classical methods estimate the noise spectrum during pauses and subtract it. Spectral subtraction sets $|\hat S|^2=\max(|Y|^2-\hat\lambda_N,\ \beta|Y|^2)$. Bins where the noise momentarily exceeds its average survive as isolated tones, the "musical noise" artefact. These methods assume the noise is stationary, which real noise rarely is.

## Masking in the STFT domain

Take the **Short-time Fourier transform (STFT)**. Then $Y(t,f)=S(t,f)+N(t,f)$ in every bin. Speech energy is sparse: at any moment it sits in a few harmonics and formants, so many bins are dominated by either speech or noise. A network (LSTM, convolutional recurrent network or U-Net) reads $|Y|$ and outputs a mask $M(t,f)$. The estimate is $\hat S=M\odot Y$, inverted by overlap-add. The training target is an **ideal** mask computed from the known $S$ and $N$ of a synthetic mixture:

$$
\mathrm{IBM}=\mathbb 1\!\Big[\tfrac{|S|^2}{|N|^2}>10^{\mathrm{LC}/10}\Big],\qquad \mathrm{IRM}=\Big(\frac{|S|^2}{|S|^2+|N|^2}\Big)^{\beta},\qquad G_{\text{Wiener}}=\frac{\xi}{1+\xi}
$$

- $\mathrm{LC}$: local criterion in dB (often 0 dB): keep a bin if speech dominates it
- $\beta$: compression of the ratio mask; $\beta=\tfrac12$ is the usual IRM
- $\xi$: a priori SNR $\lambda_S/\lambda_N$ of the bin; with oracle powers the Wiener gain is the IRM with $\beta=1$

> Figure: Two families. A: time–frequency masking. STFT, a network predicts a mask from |Y|, the mask multiplies the noisy spectrogram, and the inverse STFT reuses the noisy phase ∠Y. A complex mask (cIRM) corrects real and imaginary parts instead. B: a waveform model in Conv-TasNet's layout. A learned 1-D conv encoder (512 filters of 16 samples, stride 8, i.e. 2 ms at 8 kHz) replaces the STFT. A dilated temporal convolutional network predicts masks on that representation, and a transposed-conv decoder returns samples. Bottom: training losses and evaluation metrics.

A real-valued mask scales only the magnitude, so the output keeps the **noisy phase**. At high SNR that hardly matters. At 0 dB and below, the phase is badly wrong in many speech bins, and it caps the quality. Two fixes are common. The **complex ratio mask** (Williamson, Wang and Wang, 2016) predicts $M=S/Y$ with real and imaginary parts, which corrects phase as well. Its components are $M_r=(Y_rS_r+Y_iS_i)/|Y|^2$ and $M_i=(Y_rS_i-Y_iS_r)/|Y|^2$, compressed with a tanh for training. The other fix is to leave the STFT altogether.

## Waveform models

**Conv-TasNet** ("Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation", IEEE/ACM TASLP 2019) was built for separating speakers. It replaces the STFT with a learned encoder: a 1-D convolution with $N=512$ filters of $L=16$ samples and stride 8. A temporal convolutional network then predicts one mask per source on that representation. It stacks 8 dilated blocks (dilations 1 to 128), repeated 3 times. A transposed convolution decodes back to samples. There is no phase to estimate, because the basis is learned and real-valued. The **Demucs** denoiser is a causal waveform U-Net with an LSTM bottleneck. It is trained with an L1 waveform loss plus a multi-resolution STFT loss and runs in real time on a CPU.

$$
s_{\text{target}}=\frac{\langle \hat s,s\rangle}{\lVert s\rVert^2}\,s,\qquad \mathrm{SI\text{-}SDR}=10\log_{10}\frac{\lVert s_{\text{target}}\rVert^2}{\lVert \hat s-s_{\text{target}}\rVert^2}
$$

- $s_{\text{target}}$: projection of the estimate onto the clean signal; rescaling $\hat s$ does not change the score
- $\hat s-s_{\text{target}}$: everything else: residual noise, artefacts and distortion
- $\Delta\text{SI-SDR}$: improvement over the noisy input, the usual headline number (Le Roux, Wisdom, Erdogan and Hershey, ICASSP 2019)

Waveform models are usually trained on $-$SI-SDR directly. For separation, the loss is taken over the best speaker permutation, as in **Speaker diarization**'s EEND. Metrics that track human judgement are reported alongside. **PESQ** (ITU-T P.862, wideband P.862.2) compares the output with the clean reference and predicts quality on a −0.5 to 4.5 scale. **STOI** (Taal et al., 2011) predicts intelligibility from 0 to 1. **DNSMOS** predicts MOS without any reference, so it works on real recordings.

> Figure: A synthetic two-syllable voiced signal (f0 140 → 182 Hz, three formants) mixed with seeded noise, 8 kHz, 256-point STFT, hop 128. Default (white noise, 0 dB, IRM, noisy phase): SI-SDR goes from 0.09 dB to 11.19 dB (+11.09). The IBM keeps only 9.3% of the bins and reaches 11.47 dB, Wiener 12.30 dB, and spectral subtraction, which knows only the average noise power, 3.71 dB. Switching to the oracle clean phase lifts the same IRM to 17.68 dB, a 6.50 dB gap that is the cost of the noisy phase. At −5 dB, spectral subtraction gives −1.39 dB while Wiener still gives 9.09 dB.

Worked example: what each ideal mask does to one bin, by local SNR |S|²/|N|²

| Local SNR | IBM (LC 0 dB) | IRM (β = ½) | Wiener (β = 1) | IRM / Wiener attenuation |
| --- | --- | --- | --- | --- |
| +10 dB (ratio 10) | 1 | √(10/11) = 0.953 | 10/11 = 0.909 | −0.41 dB / −0.83 dB |
| 0 dB (ratio 1) | 0 | √(1/2) = 0.707 | 1/2 = 0.500 | −3.01 dB / −6.02 dB |
| −10 dB (ratio 0.1) | 0 | √(1/11) = 0.302 | 1/11 = 0.091 | −10.41 dB / −20.83 dB |

The IBM is all or nothing, so speech in noise-dominated bins is lost and the output can sound choppy. The ratio masks attenuate smoothly. With the noise power set to its true value and no cross term, $|Y|^2=|S|^2+|N|^2$, the spectral-subtraction gain $\sqrt{1-|N|^2/|Y|^2}$ is exactly the IRM. Spectral subtraction falls short only because it must use an average noise power instead of the true one, bin by bin.

**Reference points**

- **Conv-TasNet:** WSJ0-2mix (8 kHz, two speakers): SI-SNR improvement 15.3 dB with 5.1M parameters, above ideal magnitude masks
- **IRM:** β = 0.5 (Wang, Narayanan and Wang, 2014); the cIRM extends it to complex values (Williamson et al., 2016)
- **PESQ / STOI:** PESQ: ITU-T P.862 (2001), wideband P.862.2, scale −0.5 to 4.5; STOI: 0 to 1, intelligibility
- **DNS Challenge:** Microsoft's Deep Noise Suppression challenge, first at Interspeech 2020: open clean speech and noise sets, a mixing script, ITU-T P.808 crowdsourced MOS
- **VoiceBank+DEMAND:** Valentini-Botinhao et al. (2016): 28 training speakers, 824 test utterances; the standard small denoising benchmark
- **Dereverberation:** WPE (Nakatani et al., 2010): multi-frame linear prediction in the STFT domain; neural models target the direct path plus early reflections

**Building supervised training pairs**

1. Draw a clean utterance $s$ and a noise clip $n$ from separate corpora; optionally convolve $s$ with a room impulse response $h$.
2. Pick a random SNR and scale the noise: $n\leftarrow n\sqrt{\lVert s\rVert^2/(\lVert n\rVert^2\,10^{\mathrm{SNR}/10})}$; mix $y=s+n$.
3. Mask models: compute $S,N,Y$ by STFT and the target mask (IRM, cIRM) per bin. Waveform models: the target is $s$ itself.
4. Train on random crops of a few seconds; the mixtures are generated on the fly, so the model rarely sees the same pair twice.

- **Calls and conferencing:** causal noise suppression on each device, the setting of the DNS Challenge
- **Hearing aids:** low-latency enhancement where intelligibility (STOI) matters more than polish
- **ASR front end:** denoising before **Speech-to-text**, though enhancement artefacts can also raise WER
- **Media cleanup:** offline removal of hum, wind and reverberation from recordings, without latency limits

```python
import numpy as np, soundfile as sf
from pesq import pesq
from pystoi import stoi

def stft(x, n_fft=512, hop=256):
    w = np.sqrt(np.hanning(n_fft + 1)[:-1])            # periodic sqrt-Hann: perfect reconstruction at 50 %
    x = np.pad(x, (hop, hop + n_fft))
    return np.fft.rfft(np.lib.stride_tricks.sliding_window_view(x, n_fft)[::hop] * w, axis=-1)

def istft(X, length, n_fft=512, hop=256):
    w = np.sqrt(np.hanning(n_fft + 1)[:-1])
    frames = np.fft.irfft(X, n_fft, axis=-1) * w
    y = np.zeros(hop * (len(frames) - 1) + n_fft)
    for k, f in enumerate(frames):
        y[k * hop : k * hop + n_fft] += f
    return y[hop : hop + length]

def si_sdr(est, ref):
    t = (est @ ref) / (ref @ ref) * ref                # project onto the clean signal
    return 10 * np.log10((t @ t) / ((est - t) @ (est - t)))

s, fs = sf.read("clean.wav")                           # 16 kHz mono
n, _ = sf.read("noise.wav")
n = n[: len(s)] * np.sqrt((s @ s) / (n @ n))           # scale noise for 0 dB SNR
y = s + n
S, N, Y = stft(s), stft(n), stft(y)
irm = np.sqrt(np.abs(S) ** 2 / (np.abs(S) ** 2 + np.abs(N) ** 2 + 1e-12))
s_hat = istft(irm * Y, len(y))                         # the oracle target a network would learn

print(f"SI-SDR {si_sdr(y, s):.2f} -> {si_sdr(s_hat, s):.2f} dB")
print("PESQ-WB", pesq(fs, s, s_hat, "wb"), " STOI", stoi(s, s_hat, fs))
```

Ideal ratio mask with a perfect-reconstruction STFT, plus SI-SDR, PESQ and STOI (the numpy part was run on synthetic signals; the soundfile, pesq and pystoi calls were not)

> **Oracle masks are ceilings, not results:** IBM, IRM and Wiener numbers computed from the true $S$ and $N$, like those in the viz, are upper bounds for a model that predicts that mask with noisy phase. A trained network gets less. Also, a high SI-SDR does not guarantee a high PESQ: aggressive masks remove noise but leave artefacts, and listeners dislike those more than steady residual noise.

> **Phase is the reason waveform models won:** Conv-TasNet's title says it: a learned basis beat the ideal magnitude masks on WSJ0-2mix. Any magnitude-only method is capped by the noisy phase. Complex masks, complex spectral mapping and waveform models all remove that cap.

**Question:** A bin has |S|² = |N|² (0 dB local SNR). What do the IBM with LC = 0 dB, the IRM with β = ½ and the Wiener gain output?

- A. 1, 0.5, 0.707
- B. 0, 0.707, 0.5
- C. 0.5, 0.5, 0.5
- D. 1, 1, 1

**Answer:** B. 0, 0.707, 0.5 — The IBM keeps a bin only if the ratio is strictly above $10^{0}=1$, so it outputs 0. The IRM is $\sqrt{1/2}=0.707$ (−3 dB). The Wiener gain is $1/2$ (−6 dB).

**Question:** Why does SI-SDR first project the estimate onto the clean signal?

- A. To make the metric faster to compute
- B. So that the score ignores the output's overall gain and measures only how well its shape matches the clean speech
- C. To remove reverberation
- D. Because the clean signal is unknown

**Answer:** B. So that the score ignores the output's overall gain and measures only how well its shape matches the clean speech — $s_{\text{target}}$ is the best-scaled copy of $s$. Multiplying $\hat s$ by any non-zero constant scales both $s_{\text{target}}$ and the residual equally, so the ratio is unchanged. Everything not explained by a scaled $s$ counts as error.

**Key points**

- Model: $y=s+n$ (or $h*s+n$); recover $s$, causally for real-time use.
- Masks: IBM (binary), IRM $=(|S|^2/(|S|^2+|N|^2))^{1/2}$, Wiener $=\xi/(1+\xi)$, cIRM (complex, fixes phase).
- Magnitude masks keep the noisy phase; in the viz that costs 6.50 dB of SI-SDR at 0 dB SNR.
- Waveform models: Conv-TasNet (learned 512 × 16-sample encoder, dilated TCN), Demucs (causal waveform U-Net).
- Train on −SI-SDR (or L1 + multi-resolution STFT); report ΔSI-SDR, PESQ, STOI, DNSMOS.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
