EMNLP Findings2024
Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis
Xinyu Feng, Yuming Lin, Lihua He, You Li, Liang Chang, Ya Zhou
Abstract
Multimodal Sentiment Analysis (MSA) utilizes multimodal data to infer the users' sentiment. Previous methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modality may become dominant. In this paper, we propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) for multimodal sentiment analysis. KuDA uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality. In addition, with the obtained multimodal representation, the model can further highlight the contribution of dominant modality through the correlation evaluation loss. Extensive experiments on four MSA benchmark datasets indicate that KuDA achieves state-of-the-art performance and is able to adapt to different scenarios of dominant modality. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis cites CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis cites Integrating Multimodal Information in Large Pretrained Transformers
- CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion cites Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
- CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- MDF: A Modality-Aware Disentanglement and Fusion Framework for Multimodal Sentiment Analysis cites Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
- MDF: A Modality-Aware Disentanglement and Fusion Framework for Multimodal Sentiment Analysis cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- MDF: A Modality-Aware Disentanglement and Fusion Framework for Multimodal Sentiment Analysis cites Disentangled Representation Learning for Multimodal Emotion Recognition
- Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis cites Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
- Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis cites Integrating Multimodal Information in Large Pretrained Transformers
- Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting cites Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
- Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting cites CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
- Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting cites Integrating Multimodal Information in Large Pretrained Transformers
- Disentangled Representation Learning for Multimodal Emotion Recognition cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Cited by top-tier papers8
- Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment AnalysisMiaosen Luo, Yuncheng Jiang, Sijie MaiACM MM 2025 · 7 citations
- Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via PromptingHaoyu Zhang, Yinan Zhang, Chaolong Ying, Xiaoying Tang et al.NeurIPS 2025 · 2 citations
- CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal FusionSijie Mai, Shiqin HanCVPR 2026 · 1 citation
- MDF: A Modality-Aware Disentanglement and Fusion Framework for Multimodal Sentiment AnalysisZhongquan Jian, Wenhan Lv, Yanhao Chen, Guanran Luo et al.AAAI 2026
- T-MAD: Target-driven Multimodal Alignment for Stance DetectionZhaoDan Zhang, Jin Zhang, Xueqi Cheng, Hui XuEMNLP 2025
Builds on10
- MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisDevamanyu Hazarika, Roger Zimmermann, Soujanya PoriaACM MM 2020 · 1,037 citations
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment AnalysisWenmeng Yu, Hua Xu, Ziqi Yuan, Jiele WuAAAI 2021 · 737 citations
- Integrating Multimodal Information in Large Pretrained TransformersWasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh et al.ACL 2020 · 584 citations
- CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of ModalityWenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu et al.ACL 2020 · 376 citations
- Disentangled Representation Learning for Multimodal Emotion RecognitionDingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du et al.ACM MM 2022 · 260 citations
Related papers
- Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment AnalysisYing Zeng, Sijie Mai, Haifeng HuEMNLP Findings 2021 · 27 citations
- DDSE: A Decoupled Dual-Stream Enhanced Framework for Multimodal Sentiment Analysis with Text-Centric SSMShenjie Jiang, Zhuoyu Wang, Xuecheng Wu, Hongru Ji et al.ACM MM 2025 · 4 citations
- DiffuFuse: Diffusion-Driven Dual-Stream Fusion Framework for Multimodal Sentiment AnalysisXiongjian Lv, Yimin Wen, Hang YuACM MM 2025
- TF-Mamba: Text-enhanced Fusion Mamba with Missing Modalities for Robust Multimodal Sentiment AnalysisXiang Li, Xianfu Cheng, Dezhuang Miao, Xiaoming Zhang et al.EMNLP Findings 2025 · 9 citations
- Cross-modality Representation Interactive Learning for Multimodal Sentiment AnalysisJian Huang, Yanli Ji, Yang Yang, Heng Tao ShenACM MM 2023 · 17 citations
