ACL Findings2024
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation
Xize Cheng, Rongjie Huang, Linjun Li, Zehan Wang, Tao Jin, Aoxiong Yin, Feiyang Chen, Xinyu Duan, Baoxing Huai, Zhou Zhao
Abstract
Direct speech-to-speech translation achieves high-quality results through the introduction of discrete units obtained from self-supervised learning. However, talking head translation, converting audio-visual speech (i.e., talking head video) from one language into another, still confronts several challenges compared to audio speech: (1) Existing methods invariably rely on cascading, synthesizing via both audio and text, resulting in delays and cascading errors. (2) Talking head translation has a limited set of reference frames. If the generated translation exceeds the length of the original speech, the video sequence needs to be supplemented by repeating frames, leading to jarring video transitions. In this work, we propose a model for talking head translation, TransFace, which can directly translate audio-visual speech into audio-visual speech in other languages. It consists of a speech-tounit translation model to convert audio speech into discrete units and a unit-based audio-visual speech synthesizer, Unit2Lip, to re-synthesize synchronized audio-visual speech from discrete units in parallel. Furthermore, we introduce a Bounded Duration Predictor, ensuring isometric talking head translation and preventing duplicate reference frames. Experiments demonstrate that Unit2Lip significantly improves synchronization and boosts inference speed by a factor of ×4.35 on LRS2. Additionally, TransFace achieves impressive BLEU scores of 61.93 and 47.55 for Es-En and Fr-En on LRS3-T and 100% isochronous translations. The samples are available at https: //transface-demo.github.io/
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation cites Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
- MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation cites A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
- VoxDialogue: Can Spoken Dialogue Systems Understand Information Beyond Words? cites SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
- SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing cites Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
- SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing cites A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Cited by top-tier papers3
- SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local EditingLingyu Xiong, Xize Cheng, Jintao Tan, Xianjia Wu et al.ACM MM 2024 · 10 citations
- MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual TranslationSungwoo Cho, Jeongsoo Choi, Sungnyun Kim, Se-Young YunICCV 2025
- VoxDialogue: Can Spoken Dialogue Systems Understand Information Beyond Words?Xize Cheng, Ruofan Hu, Xiaoda Yang, Jingyu Lu et al.ICLR 2025
Builds on7
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisJungil Kong, Jaehyeon Kim, Jaekyoung BaeNeurIPS 2020 · 2,890 citations
- A Lip Sync Expert Is All You Need for Speech to Lip Generation In the WildK. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C. V. JawaharACM MM 2020 · 869 citations
- Voicebox: Text-Guided Multilingual Universal Speech Generation at ScaleMatthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer et al.NeurIPS 2023 · 613 citations
- Learning Audio-Visual Speech Representation by Masked Multimodal Cluster PredictionBowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman MohamedICLR 2022 · 460 citations
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational AbilitiesDong Zhang, Shimin Li, Xin Zhang, Jun Zhan et al.EMNLP Findings 2023 · 189 citations
Related papers
- AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech RepresentationJeongsoo Choi, Se Jin Park, Minsu Kim, Yong Man RoCVPR 2024
- Seeing What You Said: Talking Face Generation Guided by a Lip Reading ExpertJiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan et al.CVPR 2023
- TranSpeech: Speech-to-Speech Translation With Bilateral PerturbationRongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren et al.ICLR 2023 · 17 citations
- Talking Head Generation with Probabilistic Audio-to-Visual Diffusion PriorsZhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang et al.ICCV 2023 · 65 citations
- UnitY: Two-pass Direct Speech-to-speech Translation with Discrete UnitsHirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen et al.ACL 2023 · 30 citations
