EMNLP Findings2020
Adaptive Feature Selection for End-to-End Speech Translation
Abstract
Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features. In this paper, we propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. We first pre-train an ASR encoder and apply AFS to dynamically estimate the importance of each encoded speech feature to ASR. A ST encoder, stacked on top of the ASR encoder, then receives the filtered features from the (frozen) ASR encoder. We take L 0 DROP (Zhang et al., 2020) as the backbone for AFS, and adapt it to sparsify speech features with respect to both temporal and feature dimensions. Results on LibriSpeech En-Fr and MuST-C benchmarks show that AFS facilitates learning of ST by pruning out ∼84% temporal features, yielding an average translation gain of ∼1.3-1.6 BLEU and a decoding speedup of ∼1.4×. In particular, AFS reduces the performance gap compared to the cascade baseline, and outperforms it on LibriSpeech En-Fr with a BLEU score of 18.56 (without data augmentation). 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement cites Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
- Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
- Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement cites Curriculum Pre-training for End-to-End Speech Translation
- STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
- STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation cites Curriculum Pre-training for End-to-End Speech Translation
- Revisiting End-to-End Speech-to-Text Translation From Scratch cites Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement
- Revisiting End-to-End Speech-to-Text Translation From Scratch cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
- Non-Parametric Domain Adaptation for End-to-End Speech Translation cites Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
- Non-Parametric Domain Adaptation for End-to-End Speech Translation cites Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement
- Non-Parametric Domain Adaptation for End-to-End Speech Translation cites Curriculum Pre-training for End-to-End Speech Translation
- Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation cites STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation
- Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
- Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation cites Curriculum Pre-training for End-to-End Speech Translation
- Curriculum Pre-training for End-to-End Speech Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
Cited by top-tier papers8
- Revisiting End-to-End Speech-to-Text Translation From ScratchBiao Zhang, Barry Haddow, Rico SennrichICML 2022 · 46 citations
- Regularizing End-to-End Speech Translation with Triangular Decomposition AgreementYichao Du, Zhirui Zhang, Weizhi Wang, Boxing Chen et al.AAAI 2022 · 25 citations
- Non-Parametric Domain Adaptation for End-to-End Speech TranslationYichao Du, Weizhi Wang, Zhirui Zhang, Boxing Chen et al.EMNLP 2022 · 10 citations
- Discrete Cross-Modal Alignment Enables Zero-Shot Speech TranslationChen Wang, Yuchen Liu, Boxing Chen, Jiajun Zhang et al.EMNLP 2022 · 3 citations
- STEMM: Self-learning with Speech-text Manifold Mixup for Speech TranslationQingkai Fang, Rong Ye, Lei Li, Yang Feng et al.ACL 2022
Builds on4
- Curriculum Pre-training for End-to-End Speech TranslationChengyi Wang, Yu Wu, Shujie Liu, Ming Zhou et al.ACL 2020 · 100 citations
- Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech TranslationChengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang et al.AAAI 2020 · 90 citations
- Synchronous Speech Recognition and Speech-to-Text Translation with Interactive DecodingYuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou et al.AAAI 2020 · 73 citations
- On Sparsifying Encoder Outputs in Sequence-to-Sequence ModelsBiao Zhang, Ivan Titov, Rico SennrichACL Findings 2021 · 1 citation
Related papers
- Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation EncodersChen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang et al.ACL 2021
- Beyond Sentence-Level End-to-End Speech Translation: Context HelpsBiao Zhang, Ivan Titov, Barry Haddow, Rico SennrichACL 2021
- AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech TranslationXingshan Zeng, Liangyou Li, Qun LiuEMNLP Findings 2023 · 5 citations
- Improving End-to-End Speech Translation by Leveraging Auxiliary Speech and Text DataYuhao Zhang, Chen Xu, Bojie Hu, Chunliang Zhang et al.AAAI 2023 · 17 citations
- Phone Features Improve Speech TranslationElizabeth Salesky, Alan W. BlackACL 2020 · 2 citations
