ACL Findings2023
Speech-to-Speech Translation for a Real-world Unwritten Language
Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri
Abstract
We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English↔Taiwanese Hokkien as a case study, and present an endto-end solution from training data collection, modeling choices to benchmark dataset release. First, we present efforts on creating human annotated data, automatically mining data from large unlabeled speech datasets, and adopting pseudo-labeling to produce weakly supervised data. On the modeling, we take advantage of recent advances in applying self-supervised discrete representations as target for prediction in S2ST and show the effectiveness of leveraging additional text supervision from Mandarin, a language similar to Hokkien, in model training. Finally, we release an S2ST benchmark set to facilitate future research in this field. 1 2
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery cites UWSpeech: Speech to Speech Translation for Unwritten Languages
- DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation cites UWSpeech: Speech to Speech Translation for Unwritten Languages
- DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation cites R-Drop: Regularized Dropout for Neural Networks
- DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation cites DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation
Cited by top-tier papers3
- DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech TranslationQingkai Fang, Yan Zhou, Yang FengNeurIPS 2023 · 22 citations
- MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units DiscoveryAngelo Ortiz Tandazo, Manel Khentout, Youssef Benchekroun, Thomas Hueber et al.ACL 2026 · 1 citation
- DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech TranslationWeiting Tan, Jingyu Zhang, Lingfeng Shen, Daniel Khashabi et al.NeurIPS 2024 · 1 citation
Builds on8
- Reducing Transformer Depth on Demand with Structured DropoutAngela Fan, Edouard Grave, Armand JoulinICLR 2020 · 695 citations
- R-Drop: Regularized Dropout for Neural NetworksXiaobo Liang, Lijun Wu, Juntao Li, Yue Wang et al.NeurIPS 2021 · 610 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- UWSpeech: Speech to Speech Translation for Unwritten LanguagesChen Zhang, Xu Tan, Yi Ren, Tao Qin et al.AAAI 2021 · 69 citations
- Multimodal and Multilingual Embeddings for Large-Scale Speech MiningPaul-Ambroise Duquenne, Hongyu Gong, Holger SchwenkNeurIPS 2021 · 43 citations
Related papers
- Direct Speech-to-Speech Translation With Discrete UnitsAnn Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu et al.ACL 2022 · 235 citations
- Simple and Effective Unsupervised Speech TranslationChanghan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov et al.ACL 2023 · 9 citations
- UnitY: Two-pass Direct Speech-to-speech Translation with Discrete UnitsHirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen et al.ACL 2023 · 30 citations
- Exploring Methods for Building Dialects-Mandarin Code-Mixing Corpora: A Case Study in Taiwanese HokkienSin-En Lu, Bo-Han Lu, Chao-Yi Lu, Richard Tzong-Han TsaiEMNLP Findings 2022 · 5 citations
- AV-TranSpeech: Audio-Visual Robust Speech-to-Speech TranslationRongjie Huang, Huadai Liu, Xize Cheng, Yi Ren et al.ACL 2023 · 9 citations
