Lune

ACL Findings2023

CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation

Yikun Lei, Zhengshan Xue, Xiaohu Zhao, Haoran Sun, Shaolin Zhu, Xiaodong Lin, Deyi Xiong

Abstract

Distilling knowledge from a high-resource task, e.g., machine translation, is an effective way to alleviate the data scarcity problem of endto-end speech translation. However, previous works simply use the classical knowledge distillation that does not allow for adequate transfer of knowledge from machine translation. In this paper, we propose a comprehensive knowledge distillation framework for speech translation, CKDST, which is capable of comprehensively and effectively distilling knowledge from machine translation to speech translation from two perspectives: cross-modal contrastive representation distillation and simultaneous decoupled knowledge distillation. In the former, we leverage a contrastive learning objective to optimize the mutual information between speech and text representations for representation distillation in the encoder. In the later, we decouple the non-target class knowledge from target class knowledge for logits distillation in the decoder. Experiments on the MuST-C benchmark dataset demonstrate that our CKDST substantially improves the baseline by 1.2 BLEU on average in all translation directions, and outperforms previous state-ofthe-art end-to-end and cascaded speech translation models. The source code is available at https://github.com/ethanyklei/CKDST.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Decoupled Knowledge Distillation cites Contrastive Representation Distillation
  • Decoupled Knowledge Distillation cites On the Efficacy of Knowledge Distillation
  • Decoupled Knowledge Distillation cites Improved Knowledge Distillation via Teacher Assistant
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on16

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Improved Knowledge Distillation via Teacher AssistantSeyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine et al.AAAI 2020 · 1,361 citations
  3. Contrastive Representation DistillationYonglong Tian, Dilip Krishnan, Phillip IsolaICLR 2020 · 1,305 citations
  4. Decoupled Knowledge DistillationBorui Zhao, Quan Cui, Renjie Song, Yiyu Qiu et al.CVPR 2022 · 835 citations
  5. On the Efficacy of Knowledge DistillationJang Hyun Cho, Bharath HariharanICCV 2019 · 741 citations

Related papers

  1. CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech TranslationXiaohu Zhao, Haoran Sun, Yikun Lei, Shaolin Zhu et al.EMNLP Findings 2023 · 5 citations
  2. Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation TaskYun Tang, Juan Miguel Pino, Xian Li, Changhan Wang et al.ACL 2021
  3. FCGCL: Fine- and Coarse-Granularity Contrastive Learning for Speech TranslationHao Zhang, Nianwen Si, Yaqi Chen, Zhen Li et al.EMNLP Findings 2022 · 3 citations
  4. SimulSpeech: End-to-End Simultaneous Speech to Text TranslationYi Ren, Jinglin Liu, Xu Tan, Chen Zhang et al.ACL 2020 · 81 citations
  5. Consistency is Key: On Data-Efficient Modality Transfer in Speech TranslationHojin Lee, Changmin Lee, Seung-won HwangEMNLP Findings 2023
Dusk over the sea between two cliffs drawn in fine vertical lines