ACL Findings2021
A Text-Centered Shared-Private Framework via Cross-Modal Prediction for Multimodal Sentiment Analysis
Yang Wu, Zijie Lin, Yanyan Zhao, Bing Qin, Li-Nan Zhu
Abstract
Multimodal fusion is a core problem for multimodal sentiment analysis. Previous works usually treat all three modal features equally and implicitly explore the interactions between different modalities. In this paper, we break this kind of methods in two ways. Firstly, we observe that textual modality plays the most important role in multimodal sentiment analysis, and this can be seen from the previous works. Secondly, we observe that comparing to the textual modality, the other two kinds of nontextual modalities (visual and acoustic) can provide two kinds of semantics, shared and private semantics. The shared semantics from the other two modalities can obviously enhance the textual semantics and make the sentiment analysis model more robust, and the private semantics can be complementary to the textual semantics and meanwhile provide different views to improve the performance of sentiment analysis together with the shared semantics. Motivated by these two observations, we propose a text-centered shared-private framework (TCSP) for multimodal fusion, which consists of the cross-modal prediction and sentiment regression parts. Experiments on the MOSEI and MOSI datasets demonstrate the effectiveness of our shared-private framework, which outperforms all baselines. Furthermore, our approach provides a new way to utilize the unlabeled data for multimodal sentiment analysis.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
9 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences cites Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
- Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences cites Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
- Disentangled Representation Learning for Multimodal Emotion Recognition cites Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
- Disentangled Representation Learning for Multimodal Emotion Recognition cites Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content cites Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content cites Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content cites Disentangled Representation Learning for Multimodal Emotion Recognition
Cited by top-tier papers9
- Disentangled Representation Learning for Multimodal Emotion RecognitionDingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du et al.ACM MM 2022 · 260 citations
- Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language SequencesDingkang Yang, Haopeng Kuang, Shuai Huang, Lihua ZhangACM MM 2022 · 64 citations
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio ContentSheng Wu, Dongxiao He, Xiaobao Wang, Longbiao Wang et al.AAAI 2025 · 51 citations
- Adaptive Graph Learning for Multimodal Conversational Emotion DetectionGeng Tu, Tian Xie, Bin Liang, Hongpeng Wang et al.AAAI 2024 · 44 citations
- Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment DetectionYiwei Wei, Shaozu Yuan, Ruosong Yang, Lei Shen et al.ACL 2023 · 42 citations
Builds on3
- Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language AnalysisZhongkai Sun, Prathusha Kameswara Sarma, William A. Sethares, Yingyu LiangAAAI 2020 · 419 citations
- Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal FusionSijie Mai, Haifeng Hu, Songlong XingAAAI 2020 · 233 citations
- Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language AnalysisYao-Hung Hubert Tsai, Martin Q. Ma, Muqiao Yang, Ruslan Salakhutdinov et al.EMNLP 2020 · 2 citations
Related papers
- DiffuFuse: Diffusion-Driven Dual-Stream Fusion Framework for Multimodal Sentiment AnalysisXiongjian Lv, Yimin Wen, Hang YuACM MM 2025
- MDF: A Modality-Aware Disentanglement and Fusion Framework for Multimodal Sentiment AnalysisZhongquan Jian, Wenhan Lv, Yanhao Chen, Guanran Luo et al.AAAI 2026
- Cross-modality Representation Interactive Learning for Multimodal Sentiment AnalysisJian Huang, Yanli Ji, Yang Yang, Heng Tao ShenACM MM 2023 · 17 citations
- DDSE: A Decoupled Dual-Stream Enhanced Framework for Multimodal Sentiment Analysis with Text-Centric SSMShenjie Jiang, Zhuoyu Wang, Xuecheng Wu, Hongru Ji et al.ACM MM 2025 · 4 citations
- Tri-Subspaces Disentanglement for Multimodal Sentiment AnalysisChunlei Meng, Jiabin Luo, Zhenglin Yan, Zhenyu Yu et al.CVPR 2026 · 7 citations
