Voice conversion

Make speaker A's utterance sound like speaker B while keeping the words: extract speaker-independent content (AutoVC bottleneck, phonetic posteriorgrams, WavLM features), re-render it with B's identity, then vocode.

8 min read860 wordsadvanced0 viewsupdated 2026-10-15

Definition

Voice conversion (VC) transforms speech from a source speaker so that it sounds as if a target speaker said it. The linguistic content, and usually the timing and intonation, are kept. Neural VC separates a content representation from a speaker representation and recombines them. Key systems include AutoVC (Qian, Zhang, Chang, Yang and Hasegawa-Johnson, ICML 2019), which uses an autoencoder bottleneck, and kNN-VC (Baas, van Niekerk and Kamper, Interspeech 2023), which matches self-supervised features by nearest neighbours.

Input and output are both speech, which is what separates VC from Neural text-to-speech: there is no text, and the source already fixes the words, rhythm and melody. The hard part is that content and identity are mixed in every frame. Speaker identity lives in the vocal-tract resonances (formant positions), the pitch range and the voice quality. The same resonances also define the phonemes.

Parallel VC came first. Speakers A and B read the same sentences, dynamic time warping pairs up their frames, and a regression maps A's spectral features to B's. The classic regressor was a Gaussian mixture model (Stylianou, Cappé and Moulines, 1998; Toda, Black and Tokuda, 2007), later replaced by DNNs. Parallel data is expensive and only covers one pair of speakers. Non-parallel VC learns from ordinary recordings, which is what all the methods below do. Progress was benchmarked by the Voice Conversion Challenges of 2016, 2018 and 2020.

Separating content from speaker

keep what is said (content), replace who says it (speaker)source: speaker A“turn left”content encoderdrop speaker infoT×Dtarget: speaker Bany words, s to minspeaker encoderembedding s_Bdecoderor matcherD(content,s_B) → melor featuresvocoderHiFi-GAN“turn left”in B’s voicethree ways to get speaker-independent content without parallel dataAutoVC bottleneckE_cDC+ s_A (train) / s_Btrain: reconstruct A from(C, s_A); narrow C squeezesout speaker infophonetic posteriorgrampframes →ASR acoustic model givesP(phone | frame); a decodertrained on B renders itkNN-VC (SSL features)source frame x_tWavLM layer 6, 50 Hz;mean of k = 4 nearest Bframes (cosine) → vocoderparallel VC (older): A and B read the same sentences; DTW pairs frames; a GMM or DNN learnsthe A → B mapping. Non-parallel VC (above) needs only untranscribed speech from B.evaluate: speaker similarity to B (verifier cosine / EER) and intelligibility (ASR WER)
fig.The generic non-parallel pipeline: a content encoder removes speaker information from A's "turn left", a speaker encoder summarises any speech from B, a decoder or matcher combines the two, and a vocoder produces audio. Below are three ways to get speaker-independent content. AutoVC squeezes it through a narrow autoencoder bottleneck. Phonetic posteriorgrams use an ASR model's per-frame phone probabilities. kNN-VC takes WavLM layer-6 features (50 per second) and replaces each one by the mean of its 4 nearest frames from B.
C=Ec(XA, sA),X^A→B=D(C, sB),min⁡  E[∥X^A→A−XA∥22]+λ E[∥Ec(X^A→A,sA)−C∥1]C=E_c(X_A,\,s_A),\qquad \hat X_{A\to B}=D(C,\,s_B),\qquad \min\;\mathbb E\big[\lVert \hat X_{A\to A}-X_A\rVert_2^2\big]+\lambda\,\mathbb E\big[\lVert E_c(\hat X_{A\to A},s_A)-C\rVert_1\big]
XA, sAX_A,\,s_A
mel spectrogram of A's utterance and A's speaker embedding from a pretrained speaker-verification encoder
CC
content code: a few channels, downsampled in time. Its narrowness is what forces speaker information out
DD
decoder. AutoVC trains only on self-reconstruction (A→AA\to A) and swaps in sBs_B at conversion
λ\lambda
weight of the content-consistency term: the reconstruction should map back to the same code

AutoVC's argument is that if CC has just enough capacity for content, the decoder has no reason to read speaker identity from it, because sAs_A is supplied separately. With a bottleneck that is too wide, speaker information leaks through and the output still sounds like A. With one that is too narrow, words are lost. Because ss comes from a speaker encoder, AutoVC can convert to speakers not seen in training (zero-shot), with a WaveNet vocoder producing audio.

Phonetic posteriorgrams (PPGs) take content from a recogniser instead. Sun et al. (2016) used a speaker-independent ASR acoustic model's per-frame phone posteriors as the content. A conversion model trained only on target-speaker speech then maps PPGs to that speaker's features. Self-supervised features from HuBERT or WavLM work the same way without transcripts: Soft-VC (van Niekerk et al., 2022) predicts soft distributions over discrete HuBERT units. Other non-parallel approaches use adversarial losses, such as CycleGAN-VC (Kaneko and Kameoka, 2018) and StarGAN-VC (Kameoka et al., 2018).

kNN-VC: conversion without a conversion model

kNN-VC needs no trained mapping at all. WavLM's middle layers encode phonetic content well, and frames of the same phone from the same speaker cluster together. So for each source frame, kNN-VC looks up the most similar frames in a matching set of B's speech and replaces the frame by their average. Every output frame is then built from B's own frames, in A's order.

yt=1k∑r ∈ kNNk(xt; RB)r,d(x,r)=1−x⊤r∥x∥ ∥r∥y_t=\frac1k\sum_{r\,\in\,\mathrm{kNN}_k(x_t;\,\mathcal R_B)} r,\qquad d(x,r)=1-\frac{x^\top r}{\lVert x\rVert\,\lVert r\rVert}
xtx_t
WavLM-Large layer-6 feature of source frame tt (1024-D, one per 20 ms)
RB\mathcal R_B
matching set: all layer-6 frames of the target speaker's reference audio
kk
number of neighbours; the paper uses k=4k=4. Averaging smooths the output, but large kk mixes in other phones
dd
cosine distance; the converted features yty_t go to a HiFi-GAN vocoder trained on WavLM features

The vocoder is trained on prematched features: training inputs are themselves replaced by kNN matches from the same speaker, so the vocoder learns the slightly blurred features it will see at conversion time. Quality depends on the matching set. The paper reports that intelligibility and speaker similarity improve as the reference grows from seconds to minutes, since a short reference may not contain every phone.

feature space (toy 2-D; kNN-VC uses 1024-D WavLM layer 6)/a//e//i//o//u//s/src → tgt speakersource frames (48)target matching set (M = 150, 3.00 s at 50/s)converted (k = 4 mean; red = phone changed)frame 27: source /s/ → output /s/neighbours: /s/ /s/ /s/ /s/phone kept: 47/48 frames (97.9%)speaker axis: source -0.31 → output 0.16(target speaker's own frames: 0.31)distance to nearest real target frame: 0.044matching set covers all 6 phonescontent kept, every output is built only fromtarget-speaker frames.y_t = (1/k) Σ_(r ∈ kNN(x_t)) r, then a vocoder(HiFi-GAN on WavLM features) makes audio.per frame: source phone (top) and phone of the converted frame (bottom)aaaaaaaaaaaaiiieiiiiiiiiuuuuuuuuuuuueeeeeeeeeeeessssssssssssooooooooooooaaaaaaaaaaaaiiiiiiiiiiii
k neighbours
fig.A toy 2-D feature space with 6 phone clusters. Speakers shift every cluster along a speaker axis. Each of 48 source frames is replaced by the mean of its k nearest target frames. At the defaults (matching set M = 150 frames, i.e. 3 s at 50 frames/s, k = 4, speaker gap 0.6), 47 of 48 frames keep their phone (97.9%), the outputs lie 0.044 from the nearest real target frame, and the mean speaker-axis position moves from −0.31 (source) to +0.16 (the target's own frames: +0.31). Cut M to 60 and the matching set has no /s/, so all 6 /s/ frames turn into /a/ (42/48). k = 64 averages across phones: 44/48 kept, distance 0.483. A speaker gap of 1.0, close to the phone spacing, leaves only 32/48.
Target frameVectorcos(x, r)Distance 1 − cosRank
r1(0.9, 0.4)0.97570.02432
r2(1.0, −0.1)0.95620.04383
r3(0.2, 1.0)0.38460.61544
r4(0.8, 0.1)0.99730.00271
r5(−0.5, 0.9)−0.30481.30485
Worked example: one source frame x = (1.0, 0.2) against five target frames, cosine distance, k = 2

With k=2k=2 the output is the mean of r4 and r1: y=((0.8+0.9)/2, (0.1+0.4)/2)=(0.85, 0.25)y=\big((0.8+0.9)/2,\,(0.1+0.4)/2\big)=(0.85,\,0.25). With k=1k=1 it would be r4 itself, (0.8,0.1)(0.8, 0.1). With k=3k=3 it is (0.9, 0.1333)(0.9,\,0.1333). Note that r1 and r4 are the same Euclidean distance from xx (0.224), but cosine ranks r4 first because its direction is closer.

Reference points

AutoVC
Qian et al., ICML 2019; trained on VCTK with a pretrained speaker encoder; WaveNet vocoder; zero-shot to unseen speakers
kNN-VC
Baas, van Niekerk and Kamper, 2023; WavLM-Large layer 6 (1024-D, 50 Hz), k = 4, cosine distance, prematched HiFi-GAN vocoder; any-to-any
Content features
PPGs from an ASR model (Sun et al., 2016); HuBERT units or soft units (Soft-VC, 2022); WavLM layers
Speaker
speaker-verification embeddings (d-vectors, x-vectors) or, in kNN-VC, the raw matching set itself

Evaluation

VC is scored on two axes that pull against each other. Speaker similarity: the cosine between verifier embeddings of converted and real target speech, a listener similarity score, or the equal error rate (EER) of a verifier asked to tell converted speech from genuine speech (near 50% means indistinguishable). Intelligibility: WER or CER of a fixed ASR model on the converted speech, compared with the source. Naturalness is judged by MOS. A system that leaks source identity scores well on intelligibility and poorly on similarity, and an over-squeezed bottleneck does the opposite.

python
1import numpy as np, soundfile as sf, torch
2from transformers import WavLMModel
3
4wavlm = WavLMModel.from_pretrained("microsoft/wavlm-large").eval()
5
6def layer6(path):
7 wav, sr = sf.read(path, dtype="float32") # 16 kHz mono
8 with torch.no_grad():
9 out = wavlm(torch.from_numpy(wav)[None], output_hidden_states=True)
10 return out.hidden_states[6][0].numpy() # (T, 1024), 50 frames/s
11
12def knn_vc(src, ref, k=4):
13 s = src / np.linalg.norm(src, axis=1, keepdims=True)
14 r = ref / np.linalg.norm(ref, axis=1, keepdims=True)
15 idx = np.argsort(-(s @ r.T), axis=1)[:, :k] # k most cosine-similar target frames
16 return ref[idx].mean(axis=1) # (T, 1024)
17
18src = layer6("speaker_a.wav")
19ref = np.concatenate([layer6(p) for p in ["b_01.wav", "b_02.wav", "b_03.wav"]])
20converted = knn_vc(src, ref) # feed to a HiFi-GAN trained on prematched WavLM features
kNN-VC matching with Hugging Face WavLM. The feature extraction was not run here (needs torch and the checkpoint); the knn_vc function was run in numpy on random 1024-D features and on the worked example, returning (0.85, 0.25)

Use when

  • +Non-parallel methods learn from ordinary untranscribed speech
  • +Keeps the source's timing and intonation, which TTS must invent
  • +kNN-VC needs no training per speaker pair: any-to-any from a reference recording

Avoid when

  • −Content and identity are entangled; every method trades similarity against intelligibility
  • −Bottleneck sizes (AutoVC) are dataset-specific and need tuning
  • −Source pitch range and accent can still leak through and sound unlike the target

Check yourself

An AutoVC model reconstructs speech well, but converted speech still sounds like the source speaker. What is the most likely cause?

Check yourself

kNN-VC uses WavLM features at 50 frames per second. How many frames are in a 5-minute matching set, and why does a bigger set help?

Remember

  • ✓Voice conversion keeps content (and usually prosody) and changes identity; parallel methods need the same sentences, non-parallel ones do not.
  • ✓AutoVC: X^=D(Ec(XA,sA),sB)\hat X=D(E_c(X_A,s_A),s_B), trained only on self-reconstruction; a tuned bottleneck squeezes out speaker information.
  • ✓Content can also come from ASR phonetic posteriorgrams or self-supervised HuBERT/WavLM features.
  • ✓kNN-VC: replace each WavLM layer-6 frame with the mean of its k=4k=4 cosine-nearest target frames, then a prematched HiFi-GAN vocoder.
  • ✓Evaluate speaker similarity (verifier cosine, EER) and intelligibility (ASR WER/CER) together; get consent before converting to a real voice.
← back to Deep Learning roadmap