# Voice conversion

> Make speaker A's utterance sound like speaker B while keeping the words: extract speaker-independent content (AutoVC bottleneck, phonetic posteriorgrams, WavLM features), re-render it with B's identity, then vocode.

Source: https://www.mlfoundry.in/learn/voice-conversion  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 8 min read · updated 2026-10-15

**Voice conversion (VC):** transforms speech from a source speaker so that it sounds as if a target speaker said it. The linguistic content, and usually the timing and intonation, are kept. Neural VC separates a **content** representation from a **speaker** representation and recombines them. Key systems include AutoVC (Qian, Zhang, Chang, Yang and Hasegawa-Johnson, ICML 2019), which uses an autoencoder bottleneck, and kNN-VC (Baas, van Niekerk and Kamper, Interspeech 2023), which matches self-supervised features by nearest neighbours.

Input and output are both speech, which is what separates VC from **Neural text-to-speech**: there is no text, and the source already fixes the words, rhythm and melody. The hard part is that content and identity are mixed in every frame. Speaker identity lives in the vocal-tract resonances (formant positions), the pitch range and the voice quality. The same resonances also define the phonemes.

**Parallel VC** came first. Speakers A and B read the same sentences, dynamic time warping pairs up their frames, and a regression maps A's spectral features to B's. The classic regressor was a Gaussian mixture model (Stylianou, Cappé and Moulines, 1998; Toda, Black and Tokuda, 2007), later replaced by DNNs. Parallel data is expensive and only covers one pair of speakers. **Non-parallel VC** learns from ordinary recordings, which is what all the methods below do. Progress was benchmarked by the Voice Conversion Challenges of 2016, 2018 and 2020.

## Separating content from speaker

> Figure: The generic non-parallel pipeline: a content encoder removes speaker information from A's "turn left", a speaker encoder summarises any speech from B, a decoder or matcher combines the two, and a vocoder produces audio. Below are three ways to get speaker-independent content. AutoVC squeezes it through a narrow autoencoder bottleneck. Phonetic posteriorgrams use an ASR model's per-frame phone probabilities. kNN-VC takes WavLM layer-6 features (50 per second) and replaces each one by the mean of its 4 nearest frames from B.

$$
C=E_c(X_A,\,s_A),\qquad \hat X_{A\to B}=D(C,\,s_B),\qquad \min\;\mathbb E\big[\lVert \hat X_{A\to A}-X_A\rVert_2^2\big]+\lambda\,\mathbb E\big[\lVert E_c(\hat X_{A\to A},s_A)-C\rVert_1\big]
$$

- $X_A,\,s_A$: mel spectrogram of A's utterance and A's speaker embedding from a pretrained speaker-verification encoder
- $C$: content code: a few channels, downsampled in time. Its narrowness is what forces speaker information out
- $D$: decoder. AutoVC trains only on self-reconstruction ($A\to A$) and swaps in $s_B$ at conversion
- $\lambda$: weight of the content-consistency term: the reconstruction should map back to the same code

**AutoVC's argument** is that if $C$ has just enough capacity for content, the decoder has no reason to read speaker identity from it, because $s_A$ is supplied separately. With a bottleneck that is too wide, speaker information leaks through and the output still sounds like A. With one that is too narrow, words are lost. Because $s$ comes from a speaker encoder, AutoVC can convert to speakers not seen in training (zero-shot), with a WaveNet vocoder producing audio.

**Phonetic posteriorgrams (PPGs)** take content from a recogniser instead. Sun et al. (2016) used a speaker-independent ASR acoustic model's per-frame phone posteriors as the content. A conversion model trained only on target-speaker speech then maps PPGs to that speaker's features. **Self-supervised features** from HuBERT or WavLM work the same way without transcripts: Soft-VC (van Niekerk et al., 2022) predicts soft distributions over discrete HuBERT units. Other non-parallel approaches use adversarial losses, such as CycleGAN-VC (Kaneko and Kameoka, 2018) and StarGAN-VC (Kameoka et al., 2018).

## kNN-VC: conversion without a conversion model

kNN-VC needs no trained mapping at all. WavLM's middle layers encode phonetic content well, and frames of the same phone from the same speaker cluster together. So for each source frame, kNN-VC looks up the most similar frames in a **matching set** of B's speech and replaces the frame by their average. Every output frame is then built from B's own frames, in A's order.

$$
y_t=\frac1k\sum_{r\,\in\,\mathrm{kNN}_k(x_t;\,\mathcal R_B)} r,\qquad d(x,r)=1-\frac{x^\top r}{\lVert x\rVert\,\lVert r\rVert}
$$

- $x_t$: WavLM-Large layer-6 feature of source frame $t$ (1024-D, one per 20 ms)
- $\mathcal R_B$: matching set: all layer-6 frames of the target speaker's reference audio
- $k$: number of neighbours; the paper uses $k=4$. Averaging smooths the output, but large $k$ mixes in other phones
- $d$: cosine distance; the converted features $y_t$ go to a HiFi-GAN vocoder trained on WavLM features

The vocoder is trained on **prematched** features: training inputs are themselves replaced by kNN matches from the same speaker, so the vocoder learns the slightly blurred features it will see at conversion time. Quality depends on the matching set. The paper reports that intelligibility and speaker similarity improve as the reference grows from seconds to minutes, since a short reference may not contain every phone.

> Figure: A toy 2-D feature space with 6 phone clusters. Speakers shift every cluster along a speaker axis. Each of 48 source frames is replaced by the mean of its k nearest target frames. At the defaults (matching set M = 150 frames, i.e. 3 s at 50 frames/s, k = 4, speaker gap 0.6), 47 of 48 frames keep their phone (97.9%), the outputs lie 0.044 from the nearest real target frame, and the mean speaker-axis position moves from −0.31 (source) to +0.16 (the target's own frames: +0.31). Cut M to 60 and the matching set has no /s/, so all 6 /s/ frames turn into /a/ (42/48). k = 64 averages across phones: 44/48 kept, distance 0.483. A speaker gap of 1.0, close to the phone spacing, leaves only 32/48.

Worked example: one source frame x = (1.0, 0.2) against five target frames, cosine distance, k = 2

| Target frame | Vector | cos(x, r) | Distance 1 − cos | Rank |
| --- | --- | --- | --- | --- |
| r1 | (0.9, 0.4) | 0.9757 | 0.0243 | 2 |
| r2 | (1.0, −0.1) | 0.9562 | 0.0438 | 3 |
| r3 | (0.2, 1.0) | 0.3846 | 0.6154 | 4 |
| r4 | (0.8, 0.1) | 0.9973 | 0.0027 | 1 |
| r5 | (−0.5, 0.9) | −0.3048 | 1.3048 | 5 |

With $k=2$ the output is the mean of r4 and r1: $y=\big((0.8+0.9)/2,\,(0.1+0.4)/2\big)=(0.85,\,0.25)$. With $k=1$ it would be r4 itself, $(0.8, 0.1)$. With $k=3$ it is $(0.9,\,0.1333)$. Note that r1 and r4 are the same Euclidean distance from $x$ (0.224), but cosine ranks r4 first because its direction is closer.

**Reference points**

- **AutoVC:** Qian et al., ICML 2019; trained on VCTK with a pretrained speaker encoder; WaveNet vocoder; zero-shot to unseen speakers
- **kNN-VC:** Baas, van Niekerk and Kamper, 2023; WavLM-Large layer 6 (1024-D, 50 Hz), k = 4, cosine distance, prematched HiFi-GAN vocoder; any-to-any
- **Content features:** PPGs from an ASR model (Sun et al., 2016); HuBERT units or soft units (Soft-VC, 2022); WavLM layers
- **Speaker:** speaker-verification embeddings (d-vectors, x-vectors) or, in kNN-VC, the raw matching set itself

## Evaluation

VC is scored on two axes that pull against each other. **Speaker similarity**: the cosine between verifier embeddings of converted and real target speech, a listener similarity score, or the equal error rate (EER) of a verifier asked to tell converted speech from genuine speech (near 50% means indistinguishable). **Intelligibility**: WER or CER of a fixed ASR model on the converted speech, compared with the source. Naturalness is judged by MOS. A system that leaks source identity scores well on intelligibility and poorly on similarity, and an over-squeezed bottleneck does the opposite.

```python
import numpy as np, soundfile as sf, torch
from transformers import WavLMModel

wavlm = WavLMModel.from_pretrained("microsoft/wavlm-large").eval()

def layer6(path):
    wav, sr = sf.read(path, dtype="float32")             # 16 kHz mono
    with torch.no_grad():
        out = wavlm(torch.from_numpy(wav)[None], output_hidden_states=True)
    return out.hidden_states[6][0].numpy()               # (T, 1024), 50 frames/s

def knn_vc(src, ref, k=4):
    s = src / np.linalg.norm(src, axis=1, keepdims=True)
    r = ref / np.linalg.norm(ref, axis=1, keepdims=True)
    idx = np.argsort(-(s @ r.T), axis=1)[:, :k]           # k most cosine-similar target frames
    return ref[idx].mean(axis=1)                         # (T, 1024)

src = layer6("speaker_a.wav")
ref = np.concatenate([layer6(p) for p in ["b_01.wav", "b_02.wav", "b_03.wav"]])
converted = knn_vc(src, ref)      # feed to a HiFi-GAN trained on prematched WavLM features
```

kNN-VC matching with Hugging Face WavLM. The feature extraction was not run here (needs torch and the checkpoint); the knn_vc function was run in numpy on random 1024-D features and on the worked example, returning (0.85, 0.25)

> **Missing phones cannot be matched:** kNN-VC can only output sounds that exist in the matching set. If the reference never contains a phone, frames of that phone are replaced by the nearest other phone and the words change (in the figure, /s/ becomes /a/ when the matching set is cut to 60 frames). Use references with broad phonetic coverage, usually minutes rather than seconds.

> **Consent and spoofing:** VC is used for dubbing, privacy (anonymising a speaker), voice restoration for people who lost their voice, and entertainment. The same tools can impersonate a person in calls or fool voice authentication. The ASVspoof challenges benchmark detectors for converted and synthetic speech. Only convert to a voice whose owner has agreed, and label or watermark the output.

**Strengths**

- Non-parallel methods learn from ordinary untranscribed speech
- Keeps the source's timing and intonation, which TTS must invent
- kNN-VC needs no training per speaker pair: any-to-any from a reference recording

**Limitations**

- Content and identity are entangled; every method trades similarity against intelligibility
- Bottleneck sizes (AutoVC) are dataset-specific and need tuning
- Source pitch range and accent can still leak through and sound unlike the target

**Question:** An AutoVC model reconstructs speech well, but converted speech still sounds like the source speaker. What is the most likely cause?

- A. The bottleneck is too narrow
- B. The bottleneck is too wide, so speaker information passes through the content code and the decoder ignores s_B
- C. The vocoder is too small
- D. The speaker embedding has too many dimensions

**Answer:** B. The bottleneck is too wide, so speaker information passes through the content code and the decoder ignores s_B — AutoVC only works if the content code cannot carry identity. With spare capacity the decoder can read A's identity straight from $C$, so swapping $s_A$ for $s_B$ changes little. A too-narrow bottleneck causes the opposite failure: lost or slurred words.

**Question:** kNN-VC uses WavLM features at 50 frames per second. How many frames are in a 5-minute matching set, and why does a bigger set help?

- A. 1,500; the vocoder is retrained on it
- B. 15,000; more frames give better phonetic coverage and closer neighbours for each source frame
- C. 250; only one frame per phone is needed
- D. 15,000; it increases k automatically

**Answer:** B. 15,000; more frames give better phonetic coverage and closer neighbours for each source frame — $5\times60\times50=15{,}000$ frames. Nothing is retrained. A larger set just gives each source frame more candidates of the right phone, so its $k$ nearest are closer and come from the right cluster.

**Key points**

- Voice conversion keeps content (and usually prosody) and changes identity; parallel methods need the same sentences, non-parallel ones do not.
- AutoVC: $\hat X=D(E_c(X_A,s_A),s_B)$, trained only on self-reconstruction; a tuned bottleneck squeezes out speaker information.
- Content can also come from ASR phonetic posteriorgrams or self-supervised HuBERT/WavLM features.
- kNN-VC: replace each WavLM layer-6 frame with the mean of its $k=4$ cosine-nearest target frames, then a prematched HiFi-GAN vocoder.
- Evaluate speaker similarity (verifier cosine, EER) and intelligibility (ASR WER/CER) together; get consent before converting to a real voice.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
