ACL Findings2023
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis
Yakun Yu, Mingjun Zhao, Shiang Qi, Feiran Sun, Baoxun Wang, Weidong Guo, Xiaoli Wang, Lei Yang, Di Niu
Abstract
Multimodal Sentiment Analysis leverages multimodal signals to detect the sentiment of a speaker. Previous approaches concentrate on performing multimodal fusion and representation learning based on general knowledge obtained from pretrained models, which neglects the effect of domain-specific knowledge. In this paper, we propose Contrastive Knowledge Injection (ConKI) for multimodal sentiment analysis, where specific-knowledge representations for each modality can be learned together with general knowledge representations via knowledge injection based on an adapter architecture. In addition, ConKI uses a hierarchical contrastive learning procedure performed between knowledge types within every single modality, across modalities within each sample, and across samples to facilitate the effective learning of the proposed representations, hence improving multimodal sentiment predictions. The experiments on three popular multimodal sentiment analysis benchmarks show that ConKI outperforms all prior methods on a variety of performance metrics.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis cites PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis
- Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- Big Self-Supervised Models are Strong Semi-Supervised Learners cites A Simple Framework for Contrastive Learning of Visual Representations
- Supervised Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Graph Contrastive Learning with Augmentations cites A Simple Framework for Contrastive Learning of Visual Representations
- Prototype-as-Prompt: Multimodal Sentiment Prototypes Endowing Large Language Models the Capability to Perform Multimodal Sentiment Analysis cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Cited by top-tier papers4
- Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete DataAoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang et al.ACL 2025 · 8 citations
- PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment AnalysisKang He, Boyu Chen, Yuzhe Ding, Fei Li et al.AAAI 2026 · 1 citation
- Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment AnalysisKang He, Yuzhe Ding, Xinrong Wang, Fei Li et al.CVPR 2026 · 1 citation
- Prototype-as-Prompt: Multimodal Sentiment Prototypes Endowing Large Language Models the Capability to Perform Multimodal Sentiment AnalysisXianbing Zhao, Lan Luo, Hengyang Lu, Buzhou TangCVPR 2026
Builds on17
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
- Graph Contrastive Learning with AugmentationsYuning You, Tianlong Chen, Yongduo Sui, Ting Chen et al.NeurIPS 2020 · 3,042 citations
- Big Self-Supervised Models are Strong Semi-Supervised LearnersTing Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi et al.NeurIPS 2020 · 2,611 citations
- MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisDevamanyu Hazarika, Roger Zimmermann, Soujanya PoriaACM MM 2020 · 1,037 citations
Related papers
- Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion RepresentationCong-Duy Nguyen, Thong Nguyen, Duc Anh Vu, Anh Tuan LuuEMNLP Findings 2023 · 11 citations
- UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion RecognitionGuimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu et al.EMNLP 2022 · 206 citations
- KEBR: Knowledge Enhanced Self-Supervised Balanced Representation for Multimodal Sentiment AnalysisAoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang et al.ACM MM 2024 · 12 citations
- ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment AnalysisJiuding Yang, Yakun Yu, Di Niu, Weidong Guo et al.ACL 2023 · 135 citations
- Multimodal Contextual Interactions of Entities: A Modality Circular Fusion Approach for Link PredictionJing Yang, Shundong Yang, Yuan Gao, Jieming Yang et al.ACM MM 2024 · 7 citations
