Lune

ACL Findings2023

INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition

Eunseop Yoon, Hee Suk Yoon, John B. Harvill, Mark Hasegawa-Johnson, Chang Dong Yoo

Abstract

Automatic Speech Recognition (ASR) systems have attained unprecedented performance with large speech models pre-trained based on self-supervised speech representation learning. However, these pre-trained speech models suffer from representational bias as they tend to better represent those prominent accents (i.e., native (L1) English accent) in the pre-training speech corpus than less represented accents, resulting in a deteriorated performance for nonnative (L2) English accents. Although there have been some approaches to mitigate this issue, all of these methods require updating the pre-trained model weights. In this paper, we propose Information Theoretic Adversarial Prompt Tuning (INTapt), which introduces prompts concatenated to the original input that can re-modulate the attention of the pre-trained model such that the corresponding input resembles a native (L1) English speech without updating the backbone weights. INTapt is trained simultaneously in the following two manners: (1) adversarial training to reduce accent feature dependence between the original input and the prompt-concatenated input and (2) training to minimize CTC loss for improving ASR performance to a prompt-concatenated input. Experimental results show that INTapt improves the performance of L2 English and increases feature similarity between L2 and L1 accents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Momentum Contrast for Unsupervised Visual Representation Learning
  • Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo cites Momentum Contrast for Unsupervised Visual Representation Learning
  • C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion cites SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature DispersionHee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A. Hasegawa-Johnson et al.ICLR 2024 · 84 citations

Builds on5

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCoChaoning Zhang, Kang Zhang, Trung X. Pham, Axi Niu et al.CVPR 2022 · 40 citations
  3. SMSMix: Sense-Maintained Sentence Mixup for Word Sense DisambiguationHee Suk Yoon, Eunseop Yoon, John B. Harvill, Sunjae Yoon et al.EMNLP Findings 2022 · 4 citations
  4. Prefix-Tuning: Optimizing Continuous Prompts for GenerationXiang Lisa Li, Percy LiangACL 2021
  5. Momentum Contrast for Unsupervised Visual Representation LearningKaiming He, Haoqi Fan, Yuxin Wu, Saining Xie et al.CVPR 2020

Related papers

  1. How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition SystemsArchiki Prasad, Preethi JyothiACL 2020 · 18 citations
  2. Accented Speech Recognition With Accent-specific CodebooksDarshan Prabhu, Preethi Jyothi, Sriram Ganapathy, Vinit UnniEMNLP 2023 · 5 citations
  3. ADEPT: A DEbiasing PrompT FrameworkKe Yang, Charles Yu, Yi Ren Fung, Manling Li et al.AAAI 2023 · 40 citations
  4. Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social BiasesYingji Li, Mengnan Du, Xin Wang, Ying WangACL 2023 · 12 citations
  5. Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained ModelJoanna Hong, Se Jin Park, Yong Man RoEMNLP Findings 2023 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines