ACL Findings2024
Pushing the Limits of Zero-shot End-to-End Speech Translation
Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-jussà
Abstract
Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZEROS-WOT, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MUST-C and COVOST demonstrate our method's superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving Language and Modality Transfer in Translation by Character-level Modeling cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- Unified Speech-Text Pre-training for Speech Translation and Recognition cites On Layer Normalization in the Transformer Architecture
- Learning Shared Semantic Space for Speech-to-Text Translation cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- MoLoRA: Boosting LLM-based End-to-end Speech Translation with Mixture of Low-rank Experts cites LoRA: Low-Rank Adaptation of Large Language Models
- MoLoRA: Boosting LLM-based End-to-end Speech Translation with Mixture of Low-rank Experts cites Unified Speech-Text Pre-training for Speech Translation and Recognition
- MoLoRA: Boosting LLM-based End-to-end Speech Translation with Mixture of Low-rank Experts cites Learning Shared Semantic Space for Speech-to-Text Translation
Cited by top-tier papers2
- Improving Language and Modality Transfer in Translation by Character-level ModelingIoannis Tsiamas, David Dale, Marta R. Costa-jussàACL 2025 · 3 citations
- MoLoRA: Boosting LLM-based End-to-end Speech Translation with Mixture of Low-rank ExpertsHao Zhang, Yaqi Chen, Nianwen Si, XuKui Yang et al.AAAI 2026
Builds on24
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
- On Layer Normalization in the Transformer ArchitectureRuibin Xiong, Yunchang Yang, Di He, Kai Zheng et al.ICML 2020 · 1,388 citations
- Unified Speech-Text Pre-training for Speech Translation and RecognitionYun Tang, Hongyu Gong, Ning Dong, Changhan Wang et al.ACL 2022 · 104 citations
- Learning Shared Semantic Space for Speech-to-Text TranslationChi Han, Mingxuan Wang, Heng Ji, Lei LiACL Findings 2021 · 66 citations
Related papers
- Towards Zero-shot Learning for End-to-end Cross-modal Translation ModelsJichen Yang, Kai Fan, Minpeng Liao, Boxing Chen et al.EMNLP Findings 2023
- Pre-training for Speech Translation: CTC Meets Optimal TransportPhuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino et al.ICML 2023 · 33 citations
- T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine TranslationPaul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger SchwenkEMNLP 2022 · 8 citations
- Discrete Cross-Modal Alignment Enables Zero-Shot Speech TranslationChen Wang, Yuchen Liu, Boxing Chen, Jiajun Zhang et al.EMNLP 2022 · 3 citations
- CMOT: Cross-modal Mixup via Optimal Transport for Speech TranslationYan Zhou, Qingkai Fang, Yang FengACL 2023 · 24 citations
