Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Kang He, Yuzhe Ding, Xinrong Wang, Fei Li, Chong Teng, Donghong Ji
摘要
Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage crossmodal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and crossmodal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves stateof-the-art or competitive results and maintains strong performance under missing-modality settings. Our codes are available at https://github.com/kangverse/EBMC.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper37
- MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisDevamanyu Hazarika, Roger Zimmermann, Soujanya PoriaACM MM 2020 · 被引用 1,037 次
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment AnalysisWenmeng Yu, Hua Xu, Ziqi Yuan, Jiele WuAAAI 2021 · 被引用 737 次
- SMIL: Multimodal Learning with Severely Missing ModalityMengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov 等AAAI 2021 · 被引用 393 次
- Balanced Multimodal Learning via On-the-fly Gradient ModulationXiaokang Peng, Yake Wei, Andong Deng, Dong Wang 等CVPR 2022 · 被引用 264 次
- Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment AnalysisZiqi Yuan, Wei Li, Hua Xu, Wenmeng YuACM MM 2021 · 被引用 186 次
相关 Paper
- MDF: A Modality-Aware Disentanglement and Fusion Framework for Multimodal Sentiment AnalysisZhongquan Jian, Wenhan Lv, Yanhao Chen, Guanran Luo 等AAAI 2026
- Hyper-Modality Enhancement for Multimodal Sentiment Analysis with Missing ModalitiesYan Zhuang, Minhao Liu, Wei Bai, Yanru Zhang 等NeurIPS 2025 · 被引用 10 次
- PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment AnalysisKang He, Boyu Chen, Yuzhe Ding, Fei Li 等AAAI 2026 · 被引用 1 次
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing ModalitiesMingcheng Li, Dingkang Yang, Yuxuan Lei, Shunli Wang 等AAAI 2024 · 被引用 71 次
- TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy ModalitiesYan Zhuang, Minhao Liu, Yanru Zhang, Jiawen Deng 等AAAI 2026 · 被引用 2 次
