Lune

ACL Findings2023

QAP: A Quantum-Inspired Adaptive-Priority-Learning Model for Multimodal Emotion Recognition

Ziming Li, Yan Zhou, Yaxin Liu, Fuqing Zhu, Chuanpeng Yang, Songlin Hu

Abstract

Multimodal emotion recognition for video has gained considerable attention in recent years, in which three modalities (i.e., textual, visual and acoustic) are involved. Due to the diverse levels of informational content related to emotion, three modalities typically possess varying degrees of contribution to emotion recognition. More seriously, there might be inconsistencies between the emotion of individual modality and the video. The challenges mentioned above are caused by the inherent uncertainty of emotion. Inspired by the recent advances of quantum theory in modeling uncertainty, we make an initial attempt to design a quantum-inspired adaptive-priority-learning model (QAP) to address the challenges. Specifically, the quantum state is introduced to model modal features, which allows each modality to retain all emotional tendencies until the final classification. Additionally, we design Q-attention to orderly integrate three modalities, and then QAP learns modal priority adaptively so that modalities can provide different amounts of information based on priority. Experimental results on the IEMO-CAP and MOSEI datasets show that QAP establishes new state-of-the-art results.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment AnalysisMeng Luo, Hao Fei, Bobo Li, Shengqiong Wu et al.ACM MM 2024 · 23 citations

Builds on11

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. Integrating Multimodal Information in Large Pretrained TransformersWasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh et al.ACL 2020 · 584 citations
  3. M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech CuesTrisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera et al.AAAI 2020 · 282 citations
  4. Disentangled Representation Learning for Multimodal Emotion RecognitionDingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du et al.ACM MM 2022 · 260 citations
  5. Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment AnalysisZiqi Yuan, Wei Li, Hua Xu, Wenmeng YuACM MM 2021 · 186 citations

Related papers

  1. DQ-Former: Querying Transformer with Dynamic Modality Priority for Cognitive-aligned Multimodal Emotion Recognition in ConversationJing Ye, Xinpei ZhaoACM MM 2024 · 10 citations
  2. Quantum Cognitively Motivated Decision Fusion for Video Sentiment AnalysisDimitris Gkoumas, Qiuchi Li, Shahram Dehdashti, Massimo Melucci et al.AAAI 2021 · 37 citations
  3. What Does Your Smile Mean? Jointly Detecting Multi-Modal Sarcasm and Sentiment Using Quantum ProbabilityYaochen Liu, Yazhou Zhang, Qiuchi Li, Benyou Wang et al.EMNLP Findings 2021 · 37 citations
  4. Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion RecognitionHaozhe Yang, Xianqiang Gao, Jianlong Wu, Tian Gan et al.ACL Findings 2023 · 24 citations
  5. QENN: A Quantum Entanglement-Inspired Neural Network for Interaction and Relationship Prediction in Story VideosZijun Xu, Zhengqian Wu, Chunjie Zhang, Zhongyuan Wang, Chunxia Xiao, Chao LiangCVPR Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines