Lune

EMNLP Findings2024

Enhancing Polyglot Voices by Leveraging Cross-Lingual Fine-Tuning in Any-to-One Voice Conversion

Giuseppe Ruggiero, Matteo Testa, Jurgen Van de Walle, Luigi Di Caro

Abstract

The creation of artificial polyglot voices remains a challenging task, despite considerable progress in recent years. This paper investigates self-supervised learning for voice conversion to create native-sounding polyglot voices. We introduce a novel cross-lingual any-to-one voice conversion system that is able to preserve the source accent without the need for multilingual data from the target speaker. In addition, we show a novel cross-lingual fine-tuning strategy that further improves the accent and reduces the training data requirements. Objective and subjective evaluations with English, Spanish, French and Mandarin Chinese confirm that our approach improves on state-of-the-art methods, enhancing the speech intelligibility and overall quality of the converted speech, especially in cross-lingual scenarios. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Robust Speech Recognition via Large-Scale Weak Supervision cites Unsupervised Speech Recognition
  • Robust Speech Recognition via Large-Scale Weak Supervision cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • Unsupervised Speech Recognition cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations cites HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
  • Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  3. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisJungil Kong, Jaehyeon Kim, Jaekyoung BaeNeurIPS 2020 · 2,890 citations
  4. Unsupervised Speech RecognitionAlexei Baevski, Wei-Ning Hsu, Alexis Conneau, Michael AuliNeurIPS 2021 · 309 citations
  5. Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised RepresentationsHyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee et al.NeurIPS 2021 · 200 citations

Related papers

  1. O_O-VC: Synthetic Data-Driven One-to-One Alignment for Any-to-Any Voice ConversionHuu Tuong Tu, Huan Vu, Nguyen Tien Cuong, Dien Hy Ngo et al.EMNLP Findings 2025
  2. EZ-VC: Easy Zero-shot Any-to-Any Voice ConversionAdvait Joglekar, Divyanshu Singh, Rooshil Rohit Bhatia, Srinivasan UmeshEMNLP Findings 2025 · 1 citation
  3. Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised UnitsGallil Maimon, Yossi AdiEMNLP Findings 2023 · 9 citations
  4. Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear EquationGiuseppe Ruggiero, Matteo Testa, Jurgen Van de Walle, Luigi Di CaroACL Findings 2025
  5. SelfVC: Voice Conversion With Iterative Refinement using Self TransformationsPaarth Neekhara, Shehzeen Samarah Hussain, Rafael Valle, Boris Ginsburg et al.ICML 2024 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines