Lune

EMNLP Findings2021

Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis

Ying Zeng, Sijie Mai, Haifeng Hu

Abstract

Multimodal sentiment analysis (MSA) draws increasing attention with the availability of multimodal data. The boost in performance of MSA models is mainly hindered by two problems. On the one hand, recent MSA works mostly focus on learning cross-modal dynamics, but neglect to explore an optimal solution for unimodal networks, which determines the lower limit of MSA models. On the other hand, noisy information hidden in each modality interferes the learning of correct cross-modal dynamics. To address the above-mentioned problems, we propose a novel MSA framework Modulation Model for Multimodal Sentiment Analysis (M 3 SA) to identify the contribution of modalities and reduce the impact of noisy information, so as to better learn unimodal and cross-modal dynamics. Specifically, modulation loss is designed to modulate the loss contribution based on the confidence of individual modalities in each utterance, so as to explore an optimal update solution for each unimodal network. Besides, contrary to most existing works which fail to explicitly filter out noisy information, we devise a modality filter module to identify and filter out modality noise for the learning of correct cross-modal embedding. Extensive experiments on publicly datasets demonstrate that our approach achieves state-of-the-art performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TMMDA: A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment Analysis cites Integrating Multimodal Information in Large Pretrained Transformers
  • MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis cites Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. TMMDA: A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment AnalysisXianbing Zhao, Yixin Chen, Sicen Liu, Xuan Zang et al.WWW 2023 · 17 citations
  2. CFSum Coarse-to-Fine Contribution Network for Multimodal SummarizationMin Xiao, Junnan Zhu, Haitao Lin, Yu Zhou et al.ACL 2023 · 15 citations

Builds on5

  1. MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisDevamanyu Hazarika, Roger Zimmermann, Soujanya PoriaACM MM 2020 · 1,037 citations
  2. Integrating Multimodal Information in Large Pretrained TransformersWasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh et al.ACL 2020 · 584 citations
  3. Cross-Domain Few-Shot Classification via Learned Feature-Wise TransformationHung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, Ming-Hsuan YangICLR 2020 · 467 citations
  4. Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal FusionSijie Mai, Haifeng Hu, Songlong XingAAAI 2020 · 233 citations
  5. CM-BERT: Cross-Modal BERT for Text-Audio Sentiment AnalysisKaicheng Yang, Hua Xu, Kai GaoACM MM 2020 · 129 citations

Related papers

  1. Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment AnalysisXinyu Feng, Yuming Lin, Lihua He, You Li et al.EMNLP Findings 2024 · 53 citations
  2. Enhanced Experts with Uncertainty-Aware Routing for Multimodal Sentiment AnalysisZixian Gao, Disen Hu, Xun Jiang, Huimin Lu et al.ACM MM 2024 · 17 citations
  3. PSA-MF: Personality-Sentiment Aligned Multi-Level Fusion for Multimodal Sentiment AnalysisHeng Xie, Kang Zhu, Zhengqi Wen, Jianhua Tao et al.AAAI 2026 · 1 citation
  4. MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment AnalysisXingle Xu, Yongkang Liu, Dexian Cai, Shi Feng et al.ACL Findings 2026
  5. Transformer-based Label Set Generation for Multi-modal Multi-label Emotion DetectionXincheng Ju, Dong Zhang, Junhui Li, Guodong ZhouACM MM 2020 · 64 citations
Dusk over the sea between two cliffs drawn in fine vertical lines