Lune

EMNLP Findings2023

AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation

Xingshan Zeng, Liangyou Li, Qun Liu

Abstract

To alleviate the data scarcity problem in Endto-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training cites Unified Speech-Text Pre-training for Speech Translation and Recognition
  • Learning Shared Semantic Space for Speech-to-Text Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
  • Consecutive Decoding for Speech-to-text Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. Unified Speech-Text Pre-training for Speech Translation and RecognitionYun Tang, Hongyu Gong, Ning Dong, Changhan Wang et al.ACL 2022 · 104 citations
  2. Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech TranslationChengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang et al.AAAI 2020 · 90 citations
  3. Learning Shared Semantic Space for Speech-to-Text TranslationChi Han, Mingxuan Wang, Heng Ji, Lei LiACL Findings 2021 · 66 citations
  4. Consecutive Decoding for Speech-to-text TranslationQianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu et al.AAAI 2021 · 46 citations
  5. SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingZiqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu et al.EMNLP 2022 · 38 citations

Related papers

  1. CMOT: Cross-modal Mixup via Optimal Transport for Speech TranslationYan Zhou, Qingkai Fang, Yang FengACL 2023 · 24 citations
  2. Towards Zero-shot Learning for End-to-end Cross-modal Translation ModelsJichen Yang, Kai Fan, Minpeng Liao, Boxing Chen et al.EMNLP Findings 2023
  3. Pushing the Limits of Zero-shot End-to-End Speech TranslationIoannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-jussàACL Findings 2024 · 13 citations
  4. Understanding and Bridging the Modality Gap for Speech TranslationQingkai Fang, Yang FengACL 2023 · 12 citations
  5. Improving End-to-End Speech Translation by Leveraging Auxiliary Speech and Text DataYuhao Zhang, Chen Xu, Bojie Hu, Chunliang Zhang et al.AAAI 2023 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines