Lune

ACL Findings2024

StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing

Gaoxiang Cong, Yuankai Qi, Liang Li, Amin Beheshti, Zhedong Zhang, Anton van den Hengel, Ming-Hsuan Yang, Chenggang Yan, Qingming Huang

Abstract

Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing stateof-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale cites Robust Speech Recognition via Large-Scale Weak Supervision
  • Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale cites FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  • Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale cites YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for Everyone
  • FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing cites Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
  • FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing cites Robust Speech Recognition via Large-Scale Weak Supervision
  • FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing cites FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  • FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing cites HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
  • Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion cites Robust Speech Recognition via Large-Scale Weak Supervision
  • AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation cites Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion
  • YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for Everyone cites FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  • YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for Everyone cites HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
  • InstructDubber: Instruction-based Alignment for Zero-shot Movie Dubbing cites FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing
  • InstructDubber: Instruction-based Alignment for Zero-shot Movie Dubbing cites Robust Speech Recognition via Large-Scale Weak Supervision
  • InstructDubber: Instruction-based Alignment for Zero-shot Movie Dubbing cites FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  • InstructDubber: Instruction-based Alignment for Zero-shot Movie Dubbing cites HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
  • AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation cites Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
  • AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation cites Robust Speech Recognition via Large-Scale Weak Supervision
  • AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation cites FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  • AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation cites HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice ConversionYan Rong, Li LiuAAAI 2025 · 11 citations
  2. InstructDubber: Instruction-based Alignment for Zero-shot Movie DubbingZhedong Zhang, Liang Li, Gaoxiang Cong, Chunshan Liu et al.AAAI 2026 · 3 citations
  3. AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech GenerationJeongsoo Choi, Ji-Hoon Kim, Sung-Bin Kim, Tae-Hyun Oh et al.ACM MM 2025 · 3 citations
  4. FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice EnhancingGaoxiang Cong, Liang Li, Jiadong Pan, Zhedong Zhang et al.ACM MM 2025 · 2 citations
  5. AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio GenerationYan Rong, Jinting Wang, Guangzhi Lei, Shan Yang et al.ACM MM 2025 · 1 citation

Builds on12

  1. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  2. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisJungil Kong, Jaehyeon Kim, Jaekyoung BaeNeurIPS 2020 · 2,890 citations
  3. Voicebox: Text-Guided Multilingual Universal Speech Generation at ScaleMatthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer et al.NeurIPS 2023 · 613 citations
  4. YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for EveryoneEdresson Casanova, Julian Weber, Christopher Dane Shulby, Arnaldo Cândido Júnior et al.ICML 2022 · 602 citations
  5. FastSpeech 2: Fast and High-Quality End-to-End Text to SpeechYi Ren, Chenxu Hu, Xu Tan, Tao Qin et al.ICLR 2021 · 513 citations

Related papers

  1. Learning to Dub Movies via Hierarchical Prosody ModelsGaoxiang Cong, Liang Li, Yuankai Qi, Zheng-Jun Zha et al.CVPR 2023
  2. From Speaker to Dubber: Movie Dubbing with Prosody and Duration Consistency LearningZhedong Zhang, Liang Li, Gaoxiang Cong, Haibing Yin et al.ACM MM 2024 · 36 citations
  3. EmoDubber: Towards High Quality and Emotion Controllable Movie DubbingGaoxiang Cong, Jiadong Pan, Liang Li, Yuankai Qi et al.CVPR 2025
  4. V2C: Visual Voice CloningQi Chen, Mingkui Tan, Yuankai Qi, Jiaqiu Zhou et al.CVPR 2022
  5. Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie DubbingZhedong Zhang, Liang Li, Chenggang Yan, Chunshan Liu et al.CVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines