Definition
Speaker diarization splits a recording into time segments labelled by speaker, without knowing in advance who the speakers are and often not how many there are. The output is a list of (start, end, label) turns, usually stored as an RTTM file. The labels are anonymous (spk0, spk1, …), which makes diarization different from speaker identification. The classical systems cluster speaker embeddings such as x-vectors (Snyder, Garcia-Romero, Sell, Povey and Khudanpur, ICASSP 2018) or ECAPA-TDNN (Desplanques, Thienpondt and Demuynck, Interspeech 2020). End-to-end neural diarization, EEND (Fujita, Kanda, Horiguchi, Nagamatsu and Watanabe, Interspeech 2019), predicts each speaker's activity directly.
The input is a long recording, such as a meeting, a phone call or a podcast. The output is a timeline. The task is hard for four reasons. The number of speakers is unknown. Turns can be shorter than a second ("yeah", "mm-hm"). People talk over each other. Microphones, rooms and noise change what a voice sounds like. Diarization is also the step that turns a transcript from Speech-to-text into a readable "A: … B: …" dialogue.
Older systems detected speaker changes with the Bayesian information criterion (BIC) and clustered segments modelled by Gaussian mixtures, and later by i-vectors (Dehak et al., 2011). Neural embeddings replaced i-vectors around 2018. The pipeline shape stayed the same.
The modular pipeline
Voice activity detection (VAD) removes non-speech. The speech is cut into short overlapping windows; the Kaldi recipes use 1.5 s windows every 0.75 s. Each window becomes one speaker embedding. Pairs of embeddings are scored with PLDA or cosine similarity and clustered. Each cluster is one speaker. A resegmentation pass then refines the boundaries at frame level.
An x-vector network is a TDNN, a stack of 1-D convolutions (see Dilated convolution) over filterbank frames. A statistics-pooling layer then turns a variable-length window into one fixed vector. Further layers classify that vector among thousands of training speakers. Training data is augmented with noise, music, babble and reverberation. After training, the softmax is thrown away and a 512-d hidden layer is used as the embedding. ECAPA-TDNN adds squeeze-and-excitation Res2Net blocks, aggregates features from several layers, and replaces plain pooling with channel-wise attentive statistics. It is trained with an additive angular margin (AAM) softmax.
- frame-level TDNN output at frame of the window
- per-channel mean and standard deviation over time; ECAPA weights each frame by a channel-dependent attention first
- the segment-level embedding (512-d x-vector, 192-d in common ECAPA models)
Agglomerative hierarchical clustering (AHC) starts with one cluster per window and keeps merging the most similar pair. It stops when the best similarity falls below a threshold, so the threshold sets the number of speakers. Spectral clustering builds a cosine affinity matrix, prunes weak links and takes its top eigenvectors. The largest eigengap estimates the number of speakers, and k-means on the eigenvectors gives the clusters. Wang et al. (ICASSP 2018) used it with LSTM d-vectors. VBx (Landini et al., 2022) resegments with a Bayesian HMM whose states are speakers, initialised from AHC.
End-to-end: EEND and permutation-invariant training
Clustering gives each window one label, so overlapped speech is always partly missed. EEND treats diarization as multi-label frame classification. An encoder (a BLSTM in the first paper, self-attention in the follow-up by Fujita et al., ASRU 2019) emits sigmoids per frame, one per speaker. Two outputs can be on at the same time, which is how overlap is represented. The speaker order is arbitrary, so the loss is taken over the best permutation of the reference labels:
- predicted activity of each output slot at frame
- reference activity vector with the speakers reordered by permutation
- permutation-invariant training: orders (the Hungarian algorithm avoids enumerating them)
The first EEND fixed and was trained on simulated two-speaker mixtures. EEND-EDA (Horiguchi et al., Interspeech 2020) uses encoder–decoder attractors to handle a variable number of speakers. Self-attention cost grows as , and the model has a fixed number of output slots. Current systems are therefore hybrids. pyannote.audio (Bredin et al., ICASSP 2020) runs a local segmentation model on 10 s chunks with sliding windows. In version 3 that model has a powerset output: at most 3 speakers per chunk and at most 2 at once (Plaquet and Bredin, Interspeech 2023). Per-chunk speaker embeddings are then clustered so that labels agree across chunks.
Diarization error rate
- total reference speaker time: a second with two speakers counts twice
- per frame, and , summed over time
- minus correctly matched speakers, after the one-to-one label mapping that maximises overlap
Two scoring options change the number a lot. A forgiveness collar, traditionally ±0.25 s, removes the time around every reference boundary, where human labels are themselves uncertain. Overlap can be scored or excluded. DIHARD scores with no collar and with overlap included, so its DERs are much higher than old CALLHOME numbers reported with a collar and with overlap excluded.
| Term | Seconds | Where it comes from |
|---|---|---|
| Reference speaker time | 30.90 | 26.5 s of speech, of which 4.4 s has two speakers (counted twice) |
| Missed | 4.70 | 4.40 s of overlap (one label per frame) + 0.30 s of late starts / early ends |
| False alarm | 0.50 | spk2 active at 24.2–24.7 s, a pause in the reference |
| Confusion | 1.10 | B's 20.0–21.0 s put in C's cluster + 0.1 s at 12.4–12.5 s |
| DER | 20.39% | (4.70 + 0.50 + 1.10) / 30.90 = 6.30 / 30.90 |
Reference points
- x-vector
- TDNN frame layers → statistics pooling → 512-d embedding; trained as a speaker classifier with data augmentation
- ECAPA-TDNN
- SE-Res2Net blocks, multi-layer aggregation, attentive statistics pooling, AAM-softmax; 192-d embeddings in common releases
- Windows
- 1.5 s windows, 0.75 s shift in the Kaldi x-vector diarization recipes; pyannote 3 segments 10 s chunks
- Scoring
- NIST md-eval or pyannote.metrics; collar ±0.25 s traditionally; DIHARD uses no collar and scores overlap
- Data
- AMI: about 100 h of meetings, mostly 4 participants; CALLHOME: telephone calls with 2–7 speakers; DIHARD: deliberately hard, mixed domains
1from pyannote.audio import Pipeline2from pyannote.core import Annotation, Segment3from pyannote.metrics.diarization import DiarizationErrorRate45pipe = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token="hf_...")6hyp = pipe("meeting.wav", min_speakers=2, max_speakers=5) # segmentation + embeddings + clustering78for turn, _, spk in hyp.itertracks(yield_label=True):9 print(f"{turn.start:7.2f} {turn.end:7.2f} {spk}") # e.g. 12.31 15.02 SPEAKER_0110with open("meeting.rttm", "w") as f:11 hyp.write_rttm(f)1213ref = Annotation()14ref[Segment(1.0, 7.0)] = "A"15ref[Segment(5.8, 12.5)] = "B" # 1.2 s overlap with A16# collar is the TOTAL width removed around each boundary: 0.5 means ±0.25 s17der = DiarizationErrorRate(collar=0.5, skip_overlap=False)18print(f"DER = {der(ref, hyp):.2%}")Use when
- +Clustering pipeline: any number of speakers, any length; embeddings trained on huge speaker-ID data
- +EEND: handles overlap natively; optimises the diarization objective directly
- +Hybrid (pyannote, EEND + clustering): local overlap handling with global speaker consistency
Avoid when
- −Clustering pipeline: one label per window, so overlap is missed; threshold tuning sets the speaker count
- −EEND: fixed output slots, T² attention cost, needs large simulated training mixtures
- −Hybrid: more stages and hyperparameters; errors in chunk linking become confusion
Check yourself
A system finds every turn exactly but names the speakers spk2, spk0, spk1 where the reference says A, B, C. What is its DER?
Check yourself
Why does EEND need a permutation-invariant loss?
Remember
- ✓Diarization = anonymous who-spoke-when; output is RTTM turns, not names.
- ✓Modular: VAD → 1.5 s windows → x-vector (512-d) / ECAPA-TDNN (192-d) → PLDA or cosine → AHC / spectral → VBx resegmentation.
- ✓EEND: per-frame, per-speaker sigmoids handle overlap; trained with a permutation-invariant BCE, over speaker orders.
- ✓Practical systems are hybrids: local neural segmentation (pyannote: 10 s chunks, powerset) plus global embedding clustering.
- ✓DER after optimal mapping; always report the collar and the overlap rule.