Speaker diarization

Diarization answers who spoke when: embed short windows (x-vectors, ECAPA-TDNN) and cluster them, or predict per-speaker activity end to end (EEND); scored by DER = (missed + false alarm + confusion) / reference time.

8 min read884 wordscore0 viewsupdated 2026-10-15

Definition

Speaker diarization splits a recording into time segments labelled by speaker, without knowing in advance who the speakers are and often not how many there are. The output is a list of (start, end, label) turns, usually stored as an RTTM file. The labels are anonymous (spk0, spk1, …), which makes diarization different from speaker identification. The classical systems cluster speaker embeddings such as x-vectors (Snyder, Garcia-Romero, Sell, Povey and Khudanpur, ICASSP 2018) or ECAPA-TDNN (Desplanques, Thienpondt and Demuynck, Interspeech 2020). End-to-end neural diarization, EEND (Fujita, Kanda, Horiguchi, Nagamatsu and Watanabe, Interspeech 2019), predicts each speaker's activity directly.

The input is a long recording, such as a meeting, a phone call or a podcast. The output is a timeline. The task is hard for four reasons. The number of speakers is unknown. Turns can be shorter than a second ("yeah", "mm-hm"). People talk over each other. Microphones, rooms and noise change what a voice sounds like. Diarization is also the step that turns a transcript from Speech-to-text into a readable "A: … B: …" dialogue.

Older systems detected speaker changes with the Bayesian information criterion (BIC) and clustered segments modelled by Gaussian mixtures, and later by i-vectors (Dehak et al., 2011). Neural embeddings replaced i-vectors around 2018. The pipeline shape stayed the same.

The modular pipeline

Voice activity detection (VAD) removes non-speech. The speech is cut into short overlapping windows; the Kaldi recipes use 1.5 s windows every 0.75 s. Each window becomes one speaker embedding. Pairs of embeddings are scored with PLDA or cosine similarity and clustered. Each cluster is one speaker. A resegmentation pass then refines the boundaries at frame level.

"who spoke when": audio → speaker-labelled time segments (RTTM)A 3.8 sB 3.4 sC 2.1 sA 2.0 soverlap16 kHzVADspeech vsnon-speechsegment1.5 s windows0.75 s shiftembedx-vector 512-dECAPA 192-dscore pairsPLDA orcosineclusterAHC / spectral→ K speakersresegmentVBx / HMM→ RTTM50 % overlapone per windowrefineboundariesone label per window → overlap is missedend-to-end (EEND, Fujita et al. 2019): one network, multi-label outputlog-melT × F frames(subsampled)encoderBLSTM orself-attentionT × S sigmoids ŷ_t,sboth ≈ 1 → overlapPIT lossmin over φ ofΣ BCE(ŷ, y_φ)speaker order is arbitrary, so the loss takes the best of the S! label permutations
fig.Top: a 12 s toy recording with turns by A, B and C, and a 0.6 s stretch where A and B overlap. Middle: the modular pipeline. VAD, 1.5 s windows with 0.75 s shift, one x-vector (512-d) or ECAPA-TDNN (192-d) embedding per window, PLDA or cosine scoring, AHC or spectral clustering, then VBx resegmentation to an RTTM file. Each window gets a single label, so overlapped speech is missed. Bottom: EEND maps log-mel frames to T × S per-speaker sigmoids, so two outputs can be on at once. Training uses a permutation-invariant loss.

An x-vector network is a TDNN, a stack of 1-D convolutions (see Dilated convolution) over filterbank frames. A statistics-pooling layer then turns a variable-length window into one fixed vector. Further layers classify that vector among thousands of training speakers. Training data is augmented with noise, music, babble and reverberation. After training, the softmax is thrown away and a 512-d hidden layer is used as the embedding. ECAPA-TDNN adds squeeze-and-excitation Res2Net blocks, aggregates features from several layers, and replaces plain pooling with channel-wise attentive statistics. It is trained with an additive angular margin (AAM) softmax.

μ=1T∑t=1Tht,σ=1T∑t=1Tht⊙ht−μ⊙μ,e=W[μ;σ]+b\mu=\frac1T\sum_{t=1}^{T}h_t,\qquad \sigma=\sqrt{\frac1T\sum_{t=1}^{T}h_t\odot h_t-\mu\odot\mu},\qquad e=W[\mu;\sigma]+b
hth_t
frame-level TDNN output at frame tt of the window
μ, σ\mu,\ \sigma
per-channel mean and standard deviation over time; ECAPA weights each frame by a channel-dependent attention αt,c\alpha_{t,c} first
ee
the segment-level embedding (512-d x-vector, 192-d in common ECAPA models)

Agglomerative hierarchical clustering (AHC) starts with one cluster per window and keeps merging the most similar pair. It stops when the best similarity falls below a threshold, so the threshold sets the number of speakers. Spectral clustering builds a cosine affinity matrix, prunes weak links and takes its top eigenvectors. The largest eigengap estimates the number of speakers, and k-means on the eigenvectors gives the clusters. Wang et al. (ICASSP 2018) used it with LSTM d-vectors. VBx (Landini et al., 2022) resegments with a Bayesian HMM whose states are speakers, initialised from AHC.

End-to-end: EEND and permutation-invariant training

Clustering gives each window one label, so overlapped speech is always partly missed. EEND treats diarization as multi-label frame classification. An encoder (a BLSTM in the first paper, self-attention in the follow-up by Fujita et al., ASRU 2019) emits SS sigmoids per frame, one per speaker. Two outputs can be on at the same time, which is how overlap is represented. The speaker order is arbitrary, so the loss is taken over the best permutation of the reference labels:

L=1TS min⁡ϕ∈perm(S) ∑t=1TBCE(y^t, ltϕ)\mathcal L=\frac{1}{TS}\,\min_{\phi\in\mathrm{perm}(S)}\ \sum_{t=1}^{T}\mathrm{BCE}\big(\hat y_t,\ l_t^{\phi}\big)
y^t∈[0,1]S\hat y_t\in[0,1]^S
predicted activity of each output slot at frame tt
ltϕl_t^{\phi}
reference activity vector with the speakers reordered by permutation ϕ\phi
min⁡ϕ\min_\phi
permutation-invariant training: S!S! orders (the Hungarian algorithm avoids enumerating them)

The first EEND fixed S=2S=2 and was trained on simulated two-speaker mixtures. EEND-EDA (Horiguchi et al., Interspeech 2020) uses encoder–decoder attractors to handle a variable number of speakers. Self-attention cost grows as T2T^2, and the model has a fixed number of output slots. Current systems are therefore hybrids. pyannote.audio (Bredin et al., ICASSP 2020) runs a local segmentation model on 10 s chunks with sliding windows. In version 3 that model has a powerset output: at most 3 speakers per chunk and at most 2 at once (Plaquet and Bredin, Interspeech 2023). Per-chunk speaker embeddings are then clustered so that labels agree across chunks.

Diarization error rate

DER=Tmiss+TFA+TconfTref\mathrm{DER}=\frac{T_{\text{miss}}+T_{\text{FA}}+T_{\text{conf}}}{T_{\text{ref}}}
TrefT_{\text{ref}}
total reference speaker time: a second with two speakers counts twice
Tmiss, TFAT_{\text{miss}},\ T_{\text{FA}}
per frame, max⁡(0,Nref−Nsys)\max(0, N_{\text{ref}}-N_{\text{sys}}) and max⁡(0,Nsys−Nref)\max(0, N_{\text{sys}}-N_{\text{ref}}), summed over time
TconfT_{\text{conf}}
min⁡(Nref,Nsys)\min(N_{\text{ref}},N_{\text{sys}}) minus correctly matched speakers, after the one-to-one label mapping that maximises overlap

Two scoring options change the number a lot. A forgiveness collar, traditionally ±0.25 s, removes the time around every reference boundary, where human labels are themselves uncertain. Overlap can be scored or excluded. DIHARD scores with no collar and with overlap included, so its DERs are much higher than old CALLHOME numbers reported with a collar and with overlap excluded.

30 s toy meeting · 10 ms frames · collar ±0.00 s · overlap scored0 s5 s10 s15 s20 s25 s30 sref Aref Bref Csystem (embeddings + clustering) · colour = mapped reference speakerspk0→Bspk1→Aspk2→Cerrors■ overlap in reference · ■ missed · ■ false alarm · ■ confusion · ■ not scoredscored reference speaker time = 30.90 s (over 30.00 s of wall clock)splitmissed 4.70 s · false alarm 0.50 s · confusion 1.10 sDER = (4.70 + 0.50 + 1.10) / 30.90 = 20.39%optimal mapping A=spk1, B=spk0, C=spk2 (max overlap, Hungarian in general)
system
overlapmapping
fig.A 30 s toy meeting scored at 10 ms resolution. Default (clustering system, no collar, overlap scored): 30.90 s of reference speaker time; 4.70 s missed (4.40 s of it overlap), 0.50 s false alarm and 1.10 s confusion, so DER = 20.39%. A ±0.25 s collar leaves only 24.0 s of the 30 s scored and gives 16.32%. Excluding overlap drops DER to 8.60%. The overlap-aware EEND output scores 6.80%; with the collar its DER is 3.56% at ±0.25 s but 4.47% at ±0.5 s, because the collar shrinks the denominator while its 0.8 s confusion stays. Mapping labels in order instead of optimally gives 82.85%.
TermSecondsWhere it comes from
Reference speaker time30.9026.5 s of speech, of which 4.4 s has two speakers (counted twice)
Missed4.704.40 s of overlap (one label per frame) + 0.30 s of late starts / early ends
False alarm0.50spk2 active at 24.2–24.7 s, a pause in the reference
Confusion1.10B's 20.0–21.0 s put in C's cluster + 0.1 s at 12.4–12.5 s
DER20.39%(4.70 + 0.50 + 1.10) / 30.90 = 6.30 / 30.90
Worked example: the viz's default (clustering system, no collar, overlap scored)

Reference points

x-vector
TDNN frame layers → statistics pooling → 512-d embedding; trained as a speaker classifier with data augmentation
ECAPA-TDNN
SE-Res2Net blocks, multi-layer aggregation, attentive statistics pooling, AAM-softmax; 192-d embeddings in common releases
Windows
1.5 s windows, 0.75 s shift in the Kaldi x-vector diarization recipes; pyannote 3 segments 10 s chunks
Scoring
NIST md-eval or pyannote.metrics; collar ±0.25 s traditionally; DIHARD uses no collar and scores overlap
Data
AMI: about 100 h of meetings, mostly 4 participants; CALLHOME: telephone calls with 2–7 speakers; DIHARD: deliberately hard, mixed domains
python
1from pyannote.audio import Pipeline
2from pyannote.core import Annotation, Segment
3from pyannote.metrics.diarization import DiarizationErrorRate
4
5pipe = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token="hf_...")
6hyp = pipe("meeting.wav", min_speakers=2, max_speakers=5) # segmentation + embeddings + clustering
7
8for turn, _, spk in hyp.itertracks(yield_label=True):
9 print(f"{turn.start:7.2f} {turn.end:7.2f} {spk}") # e.g. 12.31 15.02 SPEAKER_01
10with open("meeting.rttm", "w") as f:
11 hyp.write_rttm(f)
12
13ref = Annotation()
14ref[Segment(1.0, 7.0)] = "A"
15ref[Segment(5.8, 12.5)] = "B" # 1.2 s overlap with A
16# collar is the TOTAL width removed around each boundary: 0.5 means ±0.25 s
17der = DiarizationErrorRate(collar=0.5, skip_overlap=False)
18print(f"DER = {der(ref, hyp):.2%}")
Diarize with pyannote.audio 3.1 and score with pyannote.metrics (not run here: needs the gated Hugging Face model and a token)

Use when

  • +Clustering pipeline: any number of speakers, any length; embeddings trained on huge speaker-ID data
  • +EEND: handles overlap natively; optimises the diarization objective directly
  • +Hybrid (pyannote, EEND + clustering): local overlap handling with global speaker consistency

Avoid when

  • −Clustering pipeline: one label per window, so overlap is missed; threshold tuning sets the speaker count
  • −EEND: fixed output slots, T² attention cost, needs large simulated training mixtures
  • −Hybrid: more stages and hyperparameters; errors in chunk linking become confusion

Check yourself

A system finds every turn exactly but names the speakers spk2, spk0, spk1 where the reference says A, B, C. What is its DER?

Check yourself

Why does EEND need a permutation-invariant loss?

Remember

  • ✓Diarization = anonymous who-spoke-when; output is RTTM turns, not names.
  • ✓Modular: VAD → 1.5 s windows → x-vector (512-d) / ECAPA-TDNN (192-d) → PLDA or cosine → AHC / spectral → VBx resegmentation.
  • ✓EEND: per-frame, per-speaker sigmoids handle overlap; trained with a permutation-invariant BCE, min⁡ϕ\min_\phi over speaker orders.
  • ✓Practical systems are hybrids: local neural segmentation (pyannote: 10 s chunks, powerset) plus global embedding clustering.
  • ✓DER =(Tmiss+TFA+Tconf)/Tref=(T_{\text{miss}}+T_{\text{FA}}+T_{\text{conf}})/T_{\text{ref}} after optimal mapping; always report the collar and the overlap rule.
← back to Deep Learning roadmap