End-to-End Zero-Shot Voice Conversion with Location-Variable Convolutions
Wonjune Kang, Mark Hasegawa-Johnson, Deb RoyAbstract
Zero-shot voice conversion is becoming an increasingly popular research topic, as it promises the ability to transform speech to sound like any speaker. However, relatively little work has been done on end-to-end methods for this task, which are appealing because they remove the need for a separate vocoder to generate audio from intermediate features. In this work, we propose LVC-VC, an end-to-end zero-shot voice conversion model that uses location-variable convolutions (LVCs) to jointly model the conversion and speech synthesis processes. LVC-VC utilizes carefully designed input features that have disentangled content and speaker information, and it uses a neural vocoder-like architecture that utilizes LVCs to efficiently combine them and perform voice conversion while directly synthesizing time domain audio. Experiments show that our model achieves especially well balanced performance between voice style transfer and speech intelligibility compared to several baselines.
Audio Demos
This page contains voice conversion samples for LVC-VC applied to audio files from the VCTK Corpus. We provide comparisons with six other models: AdaIN-VC, AGAIN-VC, AutoVC, AutoVC-F0, Blow, and NVC-Net.
Additionally, it includes samples from a larger, improved version of the model (not described in the paper), which we call LVC-VC XL. This version of the model uses a larger channel size of 32 (rather than 16) in its LVC layers, utilizes embeddings from XLSR-53 as content features, and uses information perturbation to extract only linguistic information from them (as done in NANSY). It also uses speaker embeddings from ECAPA-TDNN rather than Fast ResNet-34. Please refer to the code for more details.
Navigation
Seen-to-Seen Voice Conversion
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p241 (Male) |
p334 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
p241 (Male) |
p228 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p247 (Male) |
p245 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
p247 (Male) |
p335 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p225 (Female) |
p247 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
p225 (Female) |
p329 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p244 (Female) |
p254 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
p244 (Female) |
p314 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
Blow |
NVC-Net |
||
LVC-VC |
LVC-VC XL |
||
Unseen-to-Seen Voice Conversion
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p279 (Male) |
p315 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p279 (Male) |
p276 (Female) |
||
AdaIN-VC |
AGAIN-VC |
||
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p363 (Male) |
p241 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p363 (Male) |
p277 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p231 (Female) |
p374 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p231 (Female) |
p318 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p308 (Female) |
p256 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p308 (Female) |
p351 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
Unseen-to-Unseen Voice Conversion
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p279 (Male) |
p363 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p279 (Male) |
p240 (Female) |
||
AdaIN-VC |
AGAIN-VC |
||
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p363 (Male) |
p311 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p363 (Male) |
p312 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p231 (Female) |
p326 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p231 (Female) |
p240 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
| Source Speaker | Target Speaker | Converted | |
|---|---|---|---|
p308 (Female) |
p311 (Male) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||
p308 (Female) |
p317 (Female) |
AdaIN-VC |
AGAIN-VC |
AutoVC |
AutoVC-F0 |
||
NVC-Net |
LVC-VC |
||
LVC-VC XL |
|||