Lune

EMNLP Findings2023

CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation

Xiaohu Zhao, Haoran Sun, Yikun Lei, Shaolin Zhu, Deyi Xiong

Abstract

Deep neural networks have demonstrated their capacity in extracting features from speech inputs. However, these features may include non-linguistic speech factors such as timbre and speaker identity, which are not directly related to translation. In this paper, we propose a content-centric speech representation disentanglement learning framework for speech translation, CCSRD, which decomposes speech representations into content representations and non-linguistic representations via representation disentanglement learning. CCSRD consists of a content encoder that encodes linguistic content information from the speech input, a non-content encoder that models non-linguistic speech features, and a disentanglement module that learns disentangled representations with a cyclic reconstructor, feature reconstructor and speaker classifier trained in a multi-task learning way. Experiments on the MuST-C benchmark dataset demonstrate that CCSRD achieves an average improvement of +0.9 BLEU in two settings across five translation directions over the baseline, outperforming state-of-the-art endto-end speech translation models and cascaded models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Revisiting End-to-End Speech-to-Text Translation From Scratch cites Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Unified Speech-Text Pre-training for Speech Translation and RecognitionYun Tang, Hongyu Gong, Ning Dong, Changhan Wang et al.ACL 2022 · 104 citations
  3. Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech TranslationChengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang et al.AAAI 2020 · 90 citations
  4. Improving Disentangled Text Representation Learning with Information-Theoretic GuidancePengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon et al.ACL 2020 · 66 citations
  5. Revisiting End-to-End Speech-to-Text Translation From ScratchBiao Zhang, Barry Haddow, Rico SennrichICML 2022 · 46 citations

Related papers

  1. CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech TranslationYikun Lei, Zhengshan Xue, Xiaohu Zhao, Haoran Sun et al.ACL Findings 2023 · 9 citations
  2. STEMM: Self-learning with Speech-text Manifold Mixup for Speech TranslationQingkai Fang, Rong Ye, Lei Li, Yang Feng et al.ACL 2022
  3. Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation TaskYun Tang, Juan Miguel Pino, Xian Li, Changhan Wang et al.ACL 2021
  4. Disentangling Voice and Content with Self-Supervision for Speaker RecognitionTianchi Liu, Kong Aik Lee, Qiongqiong Wang, Haizhou LiNeurIPS 2023 · 53 citations
  5. SpeechTripleNet: End-to-End Disentangled Speech Representation Learning for Content, Timbre and ProsodyHui Lu, Xixin Wu, Zhiyong Wu, Helen MengACM MM 2023 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines