Lune

EMNLP Findings2020

Adaptive Feature Selection for End-to-End Speech Translation

Biao Zhang, Ivan Titov, Barry Haddow, Rico Sennrich

Abstract

Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features. In this paper, we propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. We first pre-train an ASR encoder and apply AFS to dynamically estimate the importance of each encoded speech feature to ASR. A ST encoder, stacked on top of the ASR encoder, then receives the filtered features from the (frozen) ASR encoder. We take L 0 DROP (Zhang et al., 2020) as the backbone for AFS, and adapt it to sparsify speech features with respect to both temporal and feature dimensions. Results on LibriSpeech En-Fr and MuST-C benchmarks show that AFS facilitates learning of ST by pruning out ∼84% temporal features, yielding an average translation gain of ∼1.3-1.6 BLEU and a decoding speedup of ∼1.4×. In particular, AFS reduces the performance gap compared to the cascade baseline, and outperforms it on LibriSpeech En-Fr with a BLEU score of 18.56 (without data augmentation). 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

8 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement cites Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
  • Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
  • Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement cites Curriculum Pre-training for End-to-End Speech Translation
  • STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
  • STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation cites Curriculum Pre-training for End-to-End Speech Translation
  • Revisiting End-to-End Speech-to-Text Translation From Scratch cites Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement
  • Revisiting End-to-End Speech-to-Text Translation From Scratch cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
  • Non-Parametric Domain Adaptation for End-to-End Speech Translation cites Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
  • Non-Parametric Domain Adaptation for End-to-End Speech Translation cites Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement
  • Non-Parametric Domain Adaptation for End-to-End Speech Translation cites Curriculum Pre-training for End-to-End Speech Translation
  • Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation cites STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation
  • Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
  • Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation cites Curriculum Pre-training for End-to-End Speech Translation
  • Curriculum Pre-training for End-to-End Speech Translation cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers8

  1. Revisiting End-to-End Speech-to-Text Translation From ScratchBiao Zhang, Barry Haddow, Rico SennrichICML 2022 · 46 citations
  2. Regularizing End-to-End Speech Translation with Triangular Decomposition AgreementYichao Du, Zhirui Zhang, Weizhi Wang, Boxing Chen et al.AAAI 2022 · 25 citations
  3. Non-Parametric Domain Adaptation for End-to-End Speech TranslationYichao Du, Weizhi Wang, Zhirui Zhang, Boxing Chen et al.EMNLP 2022 · 10 citations
  4. Discrete Cross-Modal Alignment Enables Zero-Shot Speech TranslationChen Wang, Yuchen Liu, Boxing Chen, Jiajun Zhang et al.EMNLP 2022 · 3 citations
  5. STEMM: Self-learning with Speech-text Manifold Mixup for Speech TranslationQingkai Fang, Rong Ye, Lei Li, Yang Feng et al.ACL 2022

Builds on4

  1. Curriculum Pre-training for End-to-End Speech TranslationChengyi Wang, Yu Wu, Shujie Liu, Ming Zhou et al.ACL 2020 · 100 citations
  2. Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech TranslationChengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang et al.AAAI 2020 · 90 citations
  3. Synchronous Speech Recognition and Speech-to-Text Translation with Interactive DecodingYuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou et al.AAAI 2020 · 73 citations
  4. On Sparsifying Encoder Outputs in Sequence-to-Sequence ModelsBiao Zhang, Ivan Titov, Rico SennrichACL Findings 2021 · 1 citation

Related papers

  1. Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation EncodersChen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang et al.ACL 2021
  2. Beyond Sentence-Level End-to-End Speech Translation: Context HelpsBiao Zhang, Ivan Titov, Barry Haddow, Rico SennrichACL 2021
  3. AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech TranslationXingshan Zeng, Liangyou Li, Qun LiuEMNLP Findings 2023 · 5 citations
  4. Improving End-to-End Speech Translation by Leveraging Auxiliary Speech and Text DataYuhao Zhang, Chen Xu, Bojie Hu, Chunliang Zhang et al.AAAI 2023 · 17 citations
  5. Phone Features Improve Speech TranslationElizabeth Salesky, Alan W. BlackACL 2020 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines