Lune

EMNLP Findings2022

Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis

Iek-Heng Chu, Ziyi Chen, Xinlu Yu, Mei Han, Jing Xiao, Peng Chang

Abstract

Multimodal speech emotion recognition (SER) and sentiment analysis (SA) are important techniques for human-computer interaction. Most existing multimodal approaches utilize either shallow cross-modal fusion of pretrained features, or deep cross-modal fusion with raw features. Recently, attempts have been made to fuse pretrained feature representations in a deep fusion manner during fine-tuning stage. However, those approaches have not led to improved results, partially due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. In this work, leveraging single-modal pretrained models (RoBERTa and HuBERT), we propose a novel deeplyfused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and a stage-wise cross-modal pretraining scheme to fully facilitate the cross-modal learning. Our experiment results show that the proposed method achieves state-of-the-art results on the public IEMOCAP emotion and CMU-MOSEI sentiment datasets, exceeding the previous benchmarks by a large margin.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • i-Code: An Integrative and Composable Multimodal Learning Framework cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on4

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Don't Stop Pretraining: Adapt Language Models to Domains and TasksSuchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo et al.ACL 2020 · 93 citations
  3. i-Code: An Integrative and Composable Multimodal Learning FrameworkZiyi Yang, Yuwei Fang, Chenguang Zhu, Reid Pryzant et al.AAAI 2023 · 53 citations
  4. CTAL: Pre-training Cross-modal Transformer for Audio-and-Language RepresentationsHang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu et al.EMNLP 2021 · 9 citations

Related papers

  1. CM-BERT: Cross-Modal BERT for Text-Audio Sentiment AnalysisKaicheng Yang, Hua Xu, Kai GaoACM MM 2020 · 129 citations
  2. The MERSA Dataset and a Transformer-Based Approach for Speech Emotion RecognitionEnshi Zhang, Rafael Trujillo, Christian PoellabauerACL 2024
  3. PSA-MF: Personality-Sentiment Aligned Multi-Level Fusion for Multimodal Sentiment AnalysisHeng Xie, Kang Zhu, Zhengqi Wen, Jianhua Tao et al.AAAI 2026 · 1 citation
  4. AcFormer: An Aligned and Compact Transformer for Multimodal Sentiment AnalysisDaoming Zong, Chaoyue Ding, Baoxiang Li, Jiakui Li et al.ACM MM 2023 · 27 citations
  5. Unsupervised Domain Adaptation Integrating Transformer and Mutual Information for Cross-Corpus Speech Emotion RecognitionShiqing Zhang, Ruixin Liu, Yijiao Yang, Xiaoming Zhao et al.ACM MM 2022 · 19 citations
Dusk over the sea between two cliffs drawn in fine vertical lines