Definition
Voice conversion (VC) transforms speech from a source speaker so that it sounds as if a target speaker said it. The linguistic content, and usually the timing and intonation, are kept. Neural VC separates a content representation from a speaker representation and recombines them. Key systems include AutoVC (Qian, Zhang, Chang, Yang and Hasegawa-Johnson, ICML 2019), which uses an autoencoder bottleneck, and kNN-VC (Baas, van Niekerk and Kamper, Interspeech 2023), which matches self-supervised features by nearest neighbours.
Input and output are both speech, which is what separates VC from Neural text-to-speech: there is no text, and the source already fixes the words, rhythm and melody. The hard part is that content and identity are mixed in every frame. Speaker identity lives in the vocal-tract resonances (formant positions), the pitch range and the voice quality. The same resonances also define the phonemes.
Parallel VC came first. Speakers A and B read the same sentences, dynamic time warping pairs up their frames, and a regression maps A's spectral features to B's. The classic regressor was a Gaussian mixture model (Stylianou, Cappé and Moulines, 1998; Toda, Black and Tokuda, 2007), later replaced by DNNs. Parallel data is expensive and only covers one pair of speakers. Non-parallel VC learns from ordinary recordings, which is what all the methods below do. Progress was benchmarked by the Voice Conversion Challenges of 2016, 2018 and 2020.
Separating content from speaker
- mel spectrogram of A's utterance and A's speaker embedding from a pretrained speaker-verification encoder
- content code: a few channels, downsampled in time. Its narrowness is what forces speaker information out
- decoder. AutoVC trains only on self-reconstruction () and swaps in at conversion
- weight of the content-consistency term: the reconstruction should map back to the same code
AutoVC's argument is that if has just enough capacity for content, the decoder has no reason to read speaker identity from it, because is supplied separately. With a bottleneck that is too wide, speaker information leaks through and the output still sounds like A. With one that is too narrow, words are lost. Because comes from a speaker encoder, AutoVC can convert to speakers not seen in training (zero-shot), with a WaveNet vocoder producing audio.
Phonetic posteriorgrams (PPGs) take content from a recogniser instead. Sun et al. (2016) used a speaker-independent ASR acoustic model's per-frame phone posteriors as the content. A conversion model trained only on target-speaker speech then maps PPGs to that speaker's features. Self-supervised features from HuBERT or WavLM work the same way without transcripts: Soft-VC (van Niekerk et al., 2022) predicts soft distributions over discrete HuBERT units. Other non-parallel approaches use adversarial losses, such as CycleGAN-VC (Kaneko and Kameoka, 2018) and StarGAN-VC (Kameoka et al., 2018).
kNN-VC: conversion without a conversion model
kNN-VC needs no trained mapping at all. WavLM's middle layers encode phonetic content well, and frames of the same phone from the same speaker cluster together. So for each source frame, kNN-VC looks up the most similar frames in a matching set of B's speech and replaces the frame by their average. Every output frame is then built from B's own frames, in A's order.
- WavLM-Large layer-6 feature of source frame (1024-D, one per 20 ms)
- matching set: all layer-6 frames of the target speaker's reference audio
- number of neighbours; the paper uses . Averaging smooths the output, but large mixes in other phones
- cosine distance; the converted features go to a HiFi-GAN vocoder trained on WavLM features
The vocoder is trained on prematched features: training inputs are themselves replaced by kNN matches from the same speaker, so the vocoder learns the slightly blurred features it will see at conversion time. Quality depends on the matching set. The paper reports that intelligibility and speaker similarity improve as the reference grows from seconds to minutes, since a short reference may not contain every phone.
| Target frame | Vector | cos(x, r) | Distance 1 − cos | Rank |
|---|---|---|---|---|
| r1 | (0.9, 0.4) | 0.9757 | 0.0243 | 2 |
| r2 | (1.0, −0.1) | 0.9562 | 0.0438 | 3 |
| r3 | (0.2, 1.0) | 0.3846 | 0.6154 | 4 |
| r4 | (0.8, 0.1) | 0.9973 | 0.0027 | 1 |
| r5 | (−0.5, 0.9) | −0.3048 | 1.3048 | 5 |
With the output is the mean of r4 and r1: . With it would be r4 itself, . With it is . Note that r1 and r4 are the same Euclidean distance from (0.224), but cosine ranks r4 first because its direction is closer.
Reference points
- AutoVC
- Qian et al., ICML 2019; trained on VCTK with a pretrained speaker encoder; WaveNet vocoder; zero-shot to unseen speakers
- kNN-VC
- Baas, van Niekerk and Kamper, 2023; WavLM-Large layer 6 (1024-D, 50 Hz), k = 4, cosine distance, prematched HiFi-GAN vocoder; any-to-any
- Content features
- PPGs from an ASR model (Sun et al., 2016); HuBERT units or soft units (Soft-VC, 2022); WavLM layers
- Speaker
- speaker-verification embeddings (d-vectors, x-vectors) or, in kNN-VC, the raw matching set itself
Evaluation
VC is scored on two axes that pull against each other. Speaker similarity: the cosine between verifier embeddings of converted and real target speech, a listener similarity score, or the equal error rate (EER) of a verifier asked to tell converted speech from genuine speech (near 50% means indistinguishable). Intelligibility: WER or CER of a fixed ASR model on the converted speech, compared with the source. Naturalness is judged by MOS. A system that leaks source identity scores well on intelligibility and poorly on similarity, and an over-squeezed bottleneck does the opposite.
1import numpy as np, soundfile as sf, torch2from transformers import WavLMModel34wavlm = WavLMModel.from_pretrained("microsoft/wavlm-large").eval()56def layer6(path):7 wav, sr = sf.read(path, dtype="float32") # 16 kHz mono8 with torch.no_grad():9 out = wavlm(torch.from_numpy(wav)[None], output_hidden_states=True)10 return out.hidden_states[6][0].numpy() # (T, 1024), 50 frames/s1112def knn_vc(src, ref, k=4):13 s = src / np.linalg.norm(src, axis=1, keepdims=True)14 r = ref / np.linalg.norm(ref, axis=1, keepdims=True)15 idx = np.argsort(-(s @ r.T), axis=1)[:, :k] # k most cosine-similar target frames16 return ref[idx].mean(axis=1) # (T, 1024)1718src = layer6("speaker_a.wav")19ref = np.concatenate([layer6(p) for p in ["b_01.wav", "b_02.wav", "b_03.wav"]])20converted = knn_vc(src, ref) # feed to a HiFi-GAN trained on prematched WavLM featuresUse when
- +Non-parallel methods learn from ordinary untranscribed speech
- +Keeps the source's timing and intonation, which TTS must invent
- +kNN-VC needs no training per speaker pair: any-to-any from a reference recording
Avoid when
- −Content and identity are entangled; every method trades similarity against intelligibility
- −Bottleneck sizes (AutoVC) are dataset-specific and need tuning
- −Source pitch range and accent can still leak through and sound unlike the target
Check yourself
An AutoVC model reconstructs speech well, but converted speech still sounds like the source speaker. What is the most likely cause?
Check yourself
kNN-VC uses WavLM features at 50 frames per second. How many frames are in a 5-minute matching set, and why does a bigger set help?
Remember
- ✓Voice conversion keeps content (and usually prosody) and changes identity; parallel methods need the same sentences, non-parallel ones do not.
- ✓AutoVC: , trained only on self-reconstruction; a tuned bottleneck squeezes out speaker information.
- ✓Content can also come from ASR phonetic posteriorgrams or self-supervised HuBERT/WavLM features.
- ✓kNN-VC: replace each WavLM layer-6 frame with the mean of its cosine-nearest target frames, then a prematched HiFi-GAN vocoder.
- ✓Evaluate speaker similarity (verifier cosine, EER) and intelligibility (ASR WER/CER) together; get consent before converting to a real voice.