ACL Findings2023
Sentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis
Fan Qian, Jiqing Han, Yongjun He, Tieran Zheng, Guibin Zheng
Abstract
Multimodal Sentiment Analysis (MSA) has made great progress that benefits from extraordinary fusion scheme. However, there is a lack of labeled data, resulting in severe overfitting and poor generalization for supervised models applied in this field. In this paper, we propose Sentiment Knowledge Enhanced Selfsupervised Learning (SKESL) to capture common sentimental patterns in unlabeled videos, which facilitates further learning on limited labeled data. Specifically, with the help of sentiment knowledge and non-verbal behavior, SKESL conducts sentiment word masking and predicts fine-grained word sentiment intensity, so as to embed sentiment information at the word level into pre-trained multimodal representation. In addition, a non-verbal injection method is also proposed to integrate non-verbal information into the word semantics. Experiments on two standard benchmarks of MSA clearly show that SKESL significantly outperforms the baseline, and achieves new State-Of-The-Art (SOTA) results.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
- Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data cites Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
- Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data cites MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Cited by top-tier papers1
Builds on17
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisDevamanyu Hazarika, Roger Zimmermann, Soujanya PoriaACM MM 2020 · 1,037 citations
- K-BERT: Enabling Language Representation with Knowledge GraphWeijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang et al.AAAI 2020 · 898 citations
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment AnalysisWenmeng Yu, Hua Xu, Ziqi Yuan, Jiele WuAAAI 2021 · 737 citations
Related papers
- KEBR: Knowledge Enhanced Self-Supervised Balanced Representation for Multimodal Sentiment AnalysisAoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang et al.ACM MM 2024 · 12 citations
- SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment AnalysisHao Tian, Can Gao, Xinyan Xiao, Hao Liu et al.ACL 2020 · 265 citations
- SMILE: Infusing Spatial and Motion Semantics in Masked Video LearningFida Mohammad Thoker, Letian Jiang, Chen Zhao, Bernard GhanemCVPR 2025
- Representation Learning through Multimodal Attention and Time-Sync Comments for Affective Video Content AnalysisJicai Pan, Shangfei Wang, Lin FangACM MM 2022 · 18 citations
- MART: Masked Affective RepresenTation Learning via Masked Temporal Distribution DistillationZhicheng Zhang, Pancheng Zhao, Eunil Park, Jufeng YangCVPR 2024 · 11 citations
