Lune

EMNLP Findings2023

Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

Joanna Hong, Se Jin Park, Yong Man Ro

Abstract

We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring cites Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading
  • Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition cites Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
  • MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Learning Audio-Visual Speech Representation by Masked Multimodal Cluster PredictionBowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman MohamedICLR 2022 · 460 citations
  3. Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip ReadingMinsu Kim, Jeong Hun Yeo, Yong Man RoAAAI 2022 · 86 citations
  4. MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and RecognitionXize Cheng, Tao Jin, Rongjie Huang, Linjun Li et al.ICCV 2023 · 30 citations
  5. Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability ScoringJoanna Hong, Minsu Kim, Jeongsoo Choi, Yong Man RoCVPR 2023

Related papers

  1. Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual PromptLianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei et al.EMNLP 2022 · 27 citations
  2. Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language ModelsLifu Tu, Caiming Xiong, Yingbo ZhouEMNLP Findings 2022 · 13 citations
  3. Prompt Distribution LearningYuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu et al.CVPR 2022 · 212 citations
  4. MaPLe: Multi-modal Prompt LearningMuhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan et al.CVPR 2023
  5. Prompt-Driven Referring Image Segmentation with Instance ContrastingChao Shang, Zichen Song, Heqian Qiu, Lanxiao Wang et al.CVPR 2024 · 20 citations
Dusk over the sea between two cliffs drawn in fine vertical lines