EMNLP Findings2023
Towards Zero-shot Learning for End-to-end Cross-modal Translation Models
Jichen Yang, Kai Fan, Minpeng Liao, Boxing Chen, Zhongqiang Huang
Abstract
One of the main problems in speech translation is the mismatches between different modalities. The second problem, scarcity of parallel data covering multiple modalities, means that the end-to-end multi-modal models tend to perform worse than cascade models, although there are exceptions under favorable conditions. To address these problems, we propose an endto-end zero-shot speech translation model, connecting two pre-trained uni-modality modules via word rotator's distance. The model retains the ability of zero-shot, which is like cascade models, and also can be trained in an end-toend style to avoid error propagation. Our comprehensive experiments on the MuST-C benchmarks show that our end-to-end zero-shot approach performs better than or as well as those of the CTC-based cascade models and that our end-to-end model with supervised training also matches the latest baselines.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on9
- Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech TranslationRenjie Zheng, Jun-Kun Chen, Mingbo Ma, Liang HuangICML 2021 · 74 citations
- Learning Shared Semantic Space for Speech-to-Text TranslationChi Han, Mingxuan Wang, Heng Ji, Lei LiACL Findings 2021 · 66 citations
- Probing Multi-modal Machine Translation with Pre-trained Language ModelYawei Kong, Kai FanACL Findings 2021 · 13 citations
- T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine TranslationPaul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger SchwenkEMNLP 2022 · 8 citations
- Speech Translation and the End-to-End Promise: Taking Stock of Where We AreMatthias Sperber, Matthias PaulikACL 2020 · 7 citations
Related papers
- Pushing the Limits of Zero-shot End-to-End Speech TranslationIoannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-jussàACL Findings 2024 · 13 citations
- CMOT: Cross-modal Mixup via Optimal Transport for Speech TranslationYan Zhou, Qingkai Fang, Yang FengACL 2023 · 24 citations
- AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech TranslationXingshan Zeng, Liangyou Li, Qun LiuEMNLP Findings 2023 · 5 citations
- Discrete Cross-Modal Alignment Enables Zero-Shot Speech TranslationChen Wang, Yuchen Liu, Boxing Chen, Jiajun Zhang et al.EMNLP 2022 · 3 citations
- Rethinking and Improving Multi-task Learning for End-to-end Speech TranslationYuhao Zhang, Chen Xu, Bei Li, Hao Chen et al.EMNLP 2023 · 4 citations
