Lune

EMNLP Findings2022

Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions

Torsten Wörtwein, Lisa Sheeber, Nicholas B. Allen, Jeffrey F. Cohn, Louis-Philippe Morency

Abstract

Multimodal fusion addresses the problem of analyzing spoken words in the multimodal context, including visual expressions and prosodic cues. Even when multimodal models lead to performance improvements, it is often unclear whether bimodal and trimodal interactions are learned or whether modalities are processed independently of each other. We propose Multimodal Residual Optimization (MRO) 1 to separate unimodal, bimodal, and trimodal interactions in a multimodal model. This improves interpretability as the multimodal interaction can be quantified. Inspired by Occam's razor, the main intuition of MRO is that (simpler) unimodal contributions should be learned before learning (more complex) bimodal and trimodal interactions. For example, bimodal predictions should learn to correct the mistakes (residuals) of unimodal predictions, thereby letting the bimodal predictions focus on the remaining bimodal interactions. Empirically, we observe that MRO successfully separates unimodal, bimodal, and trimodal interactions while not degrading predictive performance. We complement our empirical results with a human perception study and observe that MRO learns multimodal interactions that align with human judgments.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion cites Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!
  • Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion cites M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
  • I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts cites MultiMoDN - Multimodal, Multi-Task, Interpretable Modular Networks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. MultiMoDN - Multimodal, Multi-Task, Interpretable Modular NetworksVinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy et al.NeurIPS 2023 · 27 citations
  2. Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality FusionIshaan Singh Rawal, Alexander Matyasko, Shantanu Jaiswal, Basura Fernando et al.ICML 2024 · 8 citations
  3. I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-ExpertsJiayi Xin, Sukwon Yun, Jie Peng, Inyoung Choi et al.ICML 2025

Builds on8

  1. MERLOT: Multimodal Neural Script Knowledge ModelsRowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu et al.NeurIPS 2021 · 463 citations
  2. Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction DetectionMichael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng et al.ICLR 2020 · 71 citations
  3. Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment AnalysisYing Zeng, Sijie Mai, Haifeng HuEMNLP Findings 2021 · 27 citations
  4. M2Lens: Visualizing and Explaining Multimodal Models for Sentiment AnalysisXingbo Wang, Jianben He, Zhihua Jin, Muqiao Yang et al.IEEE VIS 2021 · 5 citations
  5. Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!Jack Hessel, Lillian LeeEMNLP 2020 · 3 citations

Related papers

  1. Balanced Multimodal Learning via On-the-fly Gradient ModulationXiaokang Peng, Yake Wei, Andong Deng, Dong Wang et al.CVPR 2022 · 264 citations
  2. Factorize, Reconstruct, Enhance: A Unified Framework for Multimodal Sentiment AnalysisZhilu Yang, Mingcheng LiCVPR 2026
  3. Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal ReasoningYucheng Wang, Yifan Hou, Aydin Javadov, Mubashara Akhtar et al.ICLR 2026 · 3 citations
  4. EMOE: Modality-Specific Enhanced Dynamic Emotion ExpertsYiyang Fang, Wenke Huang, Guancheng Wan, Kehua Su et al.CVPR 2025
  5. Boosting Multimodal Learning via Disentangled Gradient LearningShicai Wei, Chunbo Luo, Yang LuoICCV 2025 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines