EMNLP Findings2025
O_O-VC: Synthetic Data-Driven One-to-One Alignment for Any-to-Any Voice Conversion
Huu Tuong Tu, Huan Vu, Nguyen Tien Cuong, Dien Hy Ngo, Nguyen Thi Thu Trang
Abstract
Traditional voice conversion (VC) methods typically attempt to separate speaker identity and linguistic information into distinct representations, which are then combined to reconstruct the audio.However, effectively disentangling these factors remains challenging, often leading to information loss during training.In this paper, we propose a new approach that leverages synthetic speech data generated by a highquality, pretrained multispeaker text-to-speech (TTS) model.Specifically, synthetic data pairs that share the same linguistic content but differ in speaker identity are used as input-output pairs to train the voice conversion model.This enables the model to learn a direct mapping between source and target voices, effectively capturing speaker-specific characteristics while preserving linguistic content.Additionally, we introduce a flexible training strategy for any-toany voice conversion that generalizes well to unseen speakers and new languages, enhancing adaptability and performance in zero-shot scenarios.Our experiments show that our proposed method achieves a 16.35% relative reduction in word error rate and a 5.91% improvement in speaker cosine similarity, outperforming several state-of-the-art methods.Voice conversion samples can be accessed at:
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech cites DiffWave: A Versatile Diffusion Model for Audio Synthesis
- Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme cites Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
- Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme cites DiffWave: A Versatile Diffusion Model for Audio Synthesis
Builds on8
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- DiffWave: A Versatile Diffusion Model for Audio SynthesisZhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao et al.ICLR 2021 · 1,902 citations
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-SpeechJaehyeon Kim, Jungil Kong, Juhee SonICML 2021 · 1,267 citations
- Grad-TTS: A Diffusion Probabilistic Model for Text-to-SpeechVadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova et al.ICML 2021 · 715 citations
- Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling SchemeVadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova et al.ICLR 2022 · 185 citations
Related papers
- SelfVC: Voice Conversion With Iterative Refinement using Self TransformationsPaarth Neekhara, Shehzeen Samarah Hussain, Rafael Valle, Boris Ginsburg et al.ICML 2024 · 7 citations
- Improving Zero-Shot Voice Style Transfer via Disentangled Representation LearningSiyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao et al.ICLR 2021 · 64 citations
- Enhancing Polyglot Voices by Leveraging Cross-Lingual Fine-Tuning in Any-to-One Voice ConversionGiuseppe Ruggiero, Matteo Testa, Jurgen Van de Walle, Luigi Di CaroEMNLP Findings 2024
- EZ-VC: Easy Zero-shot Any-to-Any Voice ConversionAdvait Joglekar, Divyanshu Singh, Rooshil Rohit Bhatia, Srinivasan UmeshEMNLP Findings 2025 · 1 citation
- YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for EveryoneEdresson Casanova, Julian Weber, Christopher Dane Shulby, Arnaldo Cândido Júnior et al.ICML 2022 · 602 citations
