# Speaker diarization

> Diarization answers who spoke when: embed short windows (x-vectors, ECAPA-TDNN) and cluster them, or predict per-speaker activity end to end (EEND); scored by DER = (missed + false alarm + confusion) / reference time.

Source: https://www.mlfoundry.in/learn/speaker-diarization  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Speaker diarization:** splits a recording into time segments labelled by speaker, without knowing in advance who the speakers are and often not how many there are. The output is a list of (start, end, label) turns, usually stored as an RTTM file. The labels are anonymous (spk0, spk1, …), which makes diarization different from speaker identification. The classical systems cluster speaker embeddings such as x-vectors (Snyder, Garcia-Romero, Sell, Povey and Khudanpur, ICASSP 2018) or ECAPA-TDNN (Desplanques, Thienpondt and Demuynck, Interspeech 2020). End-to-end neural diarization, EEND (Fujita, Kanda, Horiguchi, Nagamatsu and Watanabe, Interspeech 2019), predicts each speaker's activity directly.

The input is a long recording, such as a meeting, a phone call or a podcast. The output is a timeline. The task is hard for four reasons. The number of speakers is unknown. Turns can be shorter than a second ("yeah", "mm-hm"). People talk over each other. Microphones, rooms and noise change what a voice sounds like. Diarization is also the step that turns a transcript from **Speech-to-text** into a readable "A: … B: …" dialogue.

Older systems detected speaker changes with the Bayesian information criterion (BIC) and clustered segments modelled by Gaussian mixtures, and later by i-vectors (Dehak et al., 2011). Neural embeddings replaced i-vectors around 2018. The pipeline shape stayed the same.

## The modular pipeline

**Voice activity detection** (VAD) removes non-speech. The speech is cut into short overlapping windows; the Kaldi recipes use 1.5 s windows every 0.75 s. Each window becomes one **speaker embedding**. Pairs of embeddings are scored with PLDA or cosine similarity and **clustered**. Each cluster is one speaker. A **resegmentation** pass then refines the boundaries at frame level.

> Figure: Top: a 12 s toy recording with turns by A, B and C, and a 0.6 s stretch where A and B overlap. Middle: the modular pipeline. VAD, 1.5 s windows with 0.75 s shift, one x-vector (512-d) or ECAPA-TDNN (192-d) embedding per window, PLDA or cosine scoring, AHC or spectral clustering, then VBx resegmentation to an RTTM file. Each window gets a single label, so overlapped speech is missed. Bottom: EEND maps log-mel frames to T × S per-speaker sigmoids, so two outputs can be on at once. Training uses a permutation-invariant loss.

An **x-vector** network is a TDNN, a stack of 1-D convolutions (see **Dilated convolution**) over filterbank frames. A statistics-pooling layer then turns a variable-length window into one fixed vector. Further layers classify that vector among thousands of training speakers. Training data is augmented with noise, music, babble and reverberation. After training, the softmax is thrown away and a 512-d hidden layer is used as the embedding. **ECAPA-TDNN** adds squeeze-and-excitation Res2Net blocks, aggregates features from several layers, and replaces plain pooling with channel-wise attentive statistics. It is trained with an additive angular margin (AAM) softmax.

$$
\mu=\frac1T\sum_{t=1}^{T}h_t,\qquad \sigma=\sqrt{\frac1T\sum_{t=1}^{T}h_t\odot h_t-\mu\odot\mu},\qquad e=W[\mu;\sigma]+b
$$

- $h_t$: frame-level TDNN output at frame $t$ of the window
- $\mu,\ \sigma$: per-channel mean and standard deviation over time; ECAPA weights each frame by a channel-dependent attention $\alpha_{t,c}$ first
- $e$: the segment-level embedding (512-d x-vector, 192-d in common ECAPA models)

**Agglomerative hierarchical clustering** (AHC) starts with one cluster per window and keeps merging the most similar pair. It stops when the best similarity falls below a threshold, so the threshold sets the number of speakers. **Spectral clustering** builds a cosine affinity matrix, prunes weak links and takes its top eigenvectors. The largest eigengap estimates the number of speakers, and k-means on the eigenvectors gives the clusters. Wang et al. (ICASSP 2018) used it with LSTM d-vectors. **VBx** (Landini et al., 2022) resegments with a Bayesian HMM whose states are speakers, initialised from AHC.

## End-to-end: EEND and permutation-invariant training

Clustering gives each window one label, so overlapped speech is always partly missed. EEND treats diarization as **multi-label frame classification**. An encoder (a BLSTM in the first paper, self-attention in the follow-up by Fujita et al., ASRU 2019) emits $S$ sigmoids per frame, one per speaker. Two outputs can be on at the same time, which is how overlap is represented. The speaker order is arbitrary, so the loss is taken over the best permutation of the reference labels:

$$
\mathcal L=\frac{1}{TS}\,\min_{\phi\in\mathrm{perm}(S)}\ \sum_{t=1}^{T}\mathrm{BCE}\big(\hat y_t,\ l_t^{\phi}\big)
$$

- $\hat y_t\in[0,1]^S$: predicted activity of each output slot at frame $t$
- $l_t^{\phi}$: reference activity vector with the speakers reordered by permutation $\phi$
- $\min_\phi$: permutation-invariant training: $S!$ orders (the Hungarian algorithm avoids enumerating them)

The first EEND fixed $S=2$ and was trained on simulated two-speaker mixtures. EEND-EDA (Horiguchi et al., Interspeech 2020) uses encoder–decoder attractors to handle a variable number of speakers. Self-attention cost grows as $T^2$, and the model has a fixed number of output slots. Current systems are therefore hybrids. **pyannote.audio** (Bredin et al., ICASSP 2020) runs a local segmentation model on 10 s chunks with sliding windows. In version 3 that model has a powerset output: at most 3 speakers per chunk and at most 2 at once (Plaquet and Bredin, Interspeech 2023). Per-chunk speaker embeddings are then clustered so that labels agree across chunks.

## Diarization error rate

$$
\mathrm{DER}=\frac{T_{\text{miss}}+T_{\text{FA}}+T_{\text{conf}}}{T_{\text{ref}}}
$$

- $T_{\text{ref}}$: total reference speaker time: a second with two speakers counts twice
- $T_{\text{miss}},\ T_{\text{FA}}$: per frame, $\max(0, N_{\text{ref}}-N_{\text{sys}})$ and $\max(0, N_{\text{sys}}-N_{\text{ref}})$, summed over time
- $T_{\text{conf}}$: $\min(N_{\text{ref}},N_{\text{sys}})$ minus correctly matched speakers, after the one-to-one label mapping that maximises overlap

Two scoring options change the number a lot. A **forgiveness collar**, traditionally ±0.25 s, removes the time around every reference boundary, where human labels are themselves uncertain. **Overlap** can be scored or excluded. DIHARD scores with no collar and with overlap included, so its DERs are much higher than old CALLHOME numbers reported with a collar and with overlap excluded.

> Figure: A 30 s toy meeting scored at 10 ms resolution. Default (clustering system, no collar, overlap scored): 30.90 s of reference speaker time; 4.70 s missed (4.40 s of it overlap), 0.50 s false alarm and 1.10 s confusion, so DER = 20.39%. A ±0.25 s collar leaves only 24.0 s of the 30 s scored and gives 16.32%. Excluding overlap drops DER to 8.60%. The overlap-aware EEND output scores 6.80%; with the collar its DER is 3.56% at ±0.25 s but 4.47% at ±0.5 s, because the collar shrinks the denominator while its 0.8 s confusion stays. Mapping labels in order instead of optimally gives 82.85%.

Worked example: the viz's default (clustering system, no collar, overlap scored)

| Term | Seconds | Where it comes from |
| --- | --- | --- |
| Reference speaker time | 30.90 | 26.5 s of speech, of which 4.4 s has two speakers (counted twice) |
| Missed | 4.70 | 4.40 s of overlap (one label per frame) + 0.30 s of late starts / early ends |
| False alarm | 0.50 | spk2 active at 24.2–24.7 s, a pause in the reference |
| Confusion | 1.10 | B's 20.0–21.0 s put in C's cluster + 0.1 s at 12.4–12.5 s |
| DER | 20.39% | (4.70 + 0.50 + 1.10) / 30.90 = 6.30 / 30.90 |

**Reference points**

- **x-vector:** TDNN frame layers → statistics pooling → 512-d embedding; trained as a speaker classifier with data augmentation
- **ECAPA-TDNN:** SE-Res2Net blocks, multi-layer aggregation, attentive statistics pooling, AAM-softmax; 192-d embeddings in common releases
- **Windows:** 1.5 s windows, 0.75 s shift in the Kaldi x-vector diarization recipes; pyannote 3 segments 10 s chunks
- **Scoring:** NIST md-eval or pyannote.metrics; collar ±0.25 s traditionally; DIHARD uses no collar and scores overlap
- **Data:** AMI: about 100 h of meetings, mostly 4 participants; CALLHOME: telephone calls with 2–7 speakers; DIHARD: deliberately hard, mixed domains

```python
from pyannote.audio import Pipeline
from pyannote.core import Annotation, Segment
from pyannote.metrics.diarization import DiarizationErrorRate

pipe = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token="hf_...")
hyp = pipe("meeting.wav", min_speakers=2, max_speakers=5)   # segmentation + embeddings + clustering

for turn, _, spk in hyp.itertracks(yield_label=True):
    print(f"{turn.start:7.2f} {turn.end:7.2f} {spk}")       # e.g.  12.31  15.02 SPEAKER_01
with open("meeting.rttm", "w") as f:
    hyp.write_rttm(f)

ref = Annotation()
ref[Segment(1.0, 7.0)] = "A"
ref[Segment(5.8, 12.5)] = "B"                                # 1.2 s overlap with A
# collar is the TOTAL width removed around each boundary: 0.5 means ±0.25 s
der = DiarizationErrorRate(collar=0.5, skip_overlap=False)
print(f"DER = {der(ref, hyp):.2%}")
```

Diarize with pyannote.audio 3.1 and score with pyannote.metrics (not run here: needs the gated Hugging Face model and a token)

> **A DER is meaningless without its scoring setup:** The same output can score 20.4% or 8.6% depending only on whether overlap is scored, as the viz shows. A collar can also move DER either way, because it removes time from the denominator as well as errors. Before comparing systems, check the collar, the overlap rule, whether the speaker count was given to the system and whether oracle VAD was used.

> **Label names never matter, label consistency does:** The scorer finds the best one-to-one mapping, so calling A "spk1" costs nothing. What costs is splitting one speaker across two clusters or merging two speakers into one. Both show up as confusion. Like WER, DER has no upper bound: false alarms can push it above 100%.

**Strengths**

- Clustering pipeline: any number of speakers, any length; embeddings trained on huge speaker-ID data
- EEND: handles overlap natively; optimises the diarization objective directly
- Hybrid (pyannote, EEND + clustering): local overlap handling with global speaker consistency

**Limitations**

- Clustering pipeline: one label per window, so overlap is missed; threshold tuning sets the speaker count
- EEND: fixed output slots, T² attention cost, needs large simulated training mixtures
- Hybrid: more stages and hyperparameters; errors in chunk linking become confusion

**Question:** A system finds every turn exactly but names the speakers spk2, spk0, spk1 where the reference says A, B, C. What is its DER?

- A. 100%, every label is wrong
- B. About 67%, two of three labels are wrong
- C. 0%, the scorer maps labels optimally before counting confusion
- D. It cannot be scored without speaker names

**Answer:** C. 0%, the scorer maps labels optimally before counting confusion — DER first finds the one-to-one mapping between system and reference labels that maximises matched time (the Hungarian algorithm). With that mapping every frame is correct, so $T_{\text{conf}}=0$. In the viz, skipping the mapping turns a 20.39% system into 82.85%.

**Question:** Why does EEND need a permutation-invariant loss?

- A. Because sigmoids cannot be trained with cross-entropy
- B. Because output slot 1 has no fixed meaning: any speaker can land in it, so the loss must use the best matching of slots to reference speakers
- C. Because it is trained without labels
- D. To make inference faster

**Answer:** B. Because output slot 1 has no fixed meaning: any speaker can land in it, so the loss must use the best matching of slots to reference speakers — Speakers have no natural order. Fixing "slot 1 = first reference speaker" would punish a correct output that happened to use the other slot. Taking $\min_\phi$ over permutations scores the output up to relabelling, which is exactly how DER scores it.

**Key points**

- Diarization = anonymous who-spoke-when; output is RTTM turns, not names.
- Modular: VAD → 1.5 s windows → x-vector (512-d) / ECAPA-TDNN (192-d) → PLDA or cosine → AHC / spectral → VBx resegmentation.
- EEND: per-frame, per-speaker sigmoids handle overlap; trained with a permutation-invariant BCE, $\min_\phi$ over speaker orders.
- Practical systems are hybrids: local neural segmentation (pyannote: 10 s chunks, powerset) plus global embedding clustering.
- DER $=(T_{\text{miss}}+T_{\text{FA}}+T_{\text{conf}})/T_{\text{ref}}$ after optimal mapping; always report the collar and the overlap rule.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
