Lune

ACL Findings2021

A Text-Centered Shared-Private Framework via Cross-Modal Prediction for Multimodal Sentiment Analysis

Yang Wu, Zijie Lin, Yanyan Zhao, Bing Qin, Li-Nan Zhu

Abstract

Multimodal fusion is a core problem for multimodal sentiment analysis. Previous works usually treat all three modal features equally and implicitly explore the interactions between different modalities. In this paper, we break this kind of methods in two ways. Firstly, we observe that textual modality plays the most important role in multimodal sentiment analysis, and this can be seen from the previous works. Secondly, we observe that comparing to the textual modality, the other two kinds of nontextual modalities (visual and acoustic) can provide two kinds of semantics, shared and private semantics. The shared semantics from the other two modalities can obviously enhance the textual semantics and make the sentiment analysis model more robust, and the private semantics can be complementary to the textual semantics and meanwhile provide different views to improve the performance of sentiment analysis together with the shared semantics. Motivated by these two observations, we propose a text-centered shared-private framework (TCSP) for multimodal fusion, which consists of the cross-modal prediction and sentiment regression parts. Experiments on the MOSEI and MOSI datasets demonstrate the effectiveness of our shared-private framework, which outperforms all baselines. Furthermore, our approach provides a new way to utilize the unlabeled data for multimodal sentiment analysis.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences cites Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
  • Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences cites Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
  • Disentangled Representation Learning for Multimodal Emotion Recognition cites Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
  • Disentangled Representation Learning for Multimodal Emotion Recognition cites Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion
  • Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content cites Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences
  • Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content cites Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
  • Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content cites Disentangled Representation Learning for Multimodal Emotion Recognition
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Disentangled Representation Learning for Multimodal Emotion RecognitionDingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du et al.ACM MM 2022 · 260 citations
  2. Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language SequencesDingkang Yang, Haopeng Kuang, Shuai Huang, Lihua ZhangACM MM 2022 · 64 citations
  3. Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio ContentSheng Wu, Dongxiao He, Xiaobao Wang, Longbiao Wang et al.AAAI 2025 · 51 citations
  4. Adaptive Graph Learning for Multimodal Conversational Emotion DetectionGeng Tu, Tian Xie, Bin Liang, Hongpeng Wang et al.AAAI 2024 · 44 citations
  5. Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment DetectionYiwei Wei, Shaozu Yuan, Ruosong Yang, Lei Shen et al.ACL 2023 · 42 citations

Builds on3

  1. Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language AnalysisZhongkai Sun, Prathusha Kameswara Sarma, William A. Sethares, Yingyu LiangAAAI 2020 · 419 citations
  2. Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal FusionSijie Mai, Haifeng Hu, Songlong XingAAAI 2020 · 233 citations
  3. Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language AnalysisYao-Hung Hubert Tsai, Martin Q. Ma, Muqiao Yang, Ruslan Salakhutdinov et al.EMNLP 2020 · 2 citations

Related papers

  1. DiffuFuse: Diffusion-Driven Dual-Stream Fusion Framework for Multimodal Sentiment AnalysisXiongjian Lv, Yimin Wen, Hang YuACM MM 2025
  2. MDF: A Modality-Aware Disentanglement and Fusion Framework for Multimodal Sentiment AnalysisZhongquan Jian, Wenhan Lv, Yanhao Chen, Guanran Luo et al.AAAI 2026
  3. Cross-modality Representation Interactive Learning for Multimodal Sentiment AnalysisJian Huang, Yanli Ji, Yang Yang, Heng Tao ShenACM MM 2023 · 17 citations
  4. DDSE: A Decoupled Dual-Stream Enhanced Framework for Multimodal Sentiment Analysis with Text-Centric SSMShenjie Jiang, Zhuoyu Wang, Xuecheng Wu, Hongru Ji et al.ACM MM 2025 · 4 citations
  5. Tri-Subspaces Disentanglement for Multimodal Sentiment AnalysisChunlei Meng, Jiabin Luo, Zhenglin Yan, Zhenyu Yu et al.CVPR 2026 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines