Lune

EMNLP Findings2023

Debiasing Multimodal Models via Causal Information Minimization

Vaidehi Patil, Adyasha Maharana, Mohit Bansal

Abstract

Most existing debiasing methods for multimodal models, including causal intervention and inference methods, utilize approximate heuristics to represent the biases, such as shallow features from early stages of training or unimodal features for multimodal tasks like VQA, etc., which may not be accurate. In this paper, we study bias arising from confounders in a causal graph for multimodal data, and examine a novel approach that leverages causallymotivated information minimization to learn the confounder representations. Robust predictive features contain diverse information that helps a model generalize to out-of-distribution data. Hence, minimizing the information content of features obtained from a pretrained biased model helps learn the simplest predictive features that capture the underlying data distribution. We treat these features as confounder representations and use them via methods motivated by causal theory to remove bias from models. We find that the learned confounder representations indeed capture dataset biases and the proposed debiasing methods improve out-of-distribution (OOD) performance on multiple multimodal datasets without sacrificing in-distribution performance. Additionally, we introduce a novel metric to quantify the sufficiency of spurious features in models' predictions that further demonstrates the effectiveness of our proposed methods. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization cites MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Debiasing Multimodal Large Language Models via Noise-Aware Preference OptimizationZefeng Zhang, Hengzhu Tang, Jiawei Sheng, Zhenyu Zhang et al.CVPR 2025

Builds on21

  1. Large-Scale Adversarial Training for Vision-and-Language Representation LearningZhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu et al.NeurIPS 2020 · 561 citations
  2. Taking a HINT: Leveraging Explanations to Make Vision and Language Models More GroundedRamprasaath Ramasamy Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin et al.ICCV 2019 · 288 citations
  3. Domino: Discovering Systematic Errors with Cross-Modal EmbeddingsSabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck et al.ICLR 2022 · 178 citations
  4. MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question AnsweringTejas Gokhale, Pratyay Banerjee, Chitta Baral, Yezhou YangEMNLP 2020 · 136 citations
  5. Counterfactual Invariance to Spurious Correlations in Text ClassificationVictor Veitch, Alexander D'Amour, Steve Yadlowsky, Jacob EisensteinNeurIPS 2021 · 108 citations

Related papers

  1. Debiasing Concept-based Explanations with Causal AnalysisMohammad Taha Bahadori, David HeckermanICLR 2021 · 8 citations
  2. Deconfounded Multimodal Learning for Spatio-temporal Video GroundingJiawei Wang, Zhanchang Ma, Da Cao, Yuquan Le et al.ACM MM 2023 · 7 citations
  3. Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert RoutingZichen Wu, Hsiu-Yuan Huang, Yunfang WuEMNLP Findings 2025
  4. Investigating Ensemble Methods for Model Robustness Improvement of Text ClassifiersJieyu Zhao, Xuezhi Wang, Yao Qin, Jilin Chen et al.EMNLP Findings 2022
  5. Generative Interventions for Causal LearningChengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang et al.CVPR 2021
Dusk over the sea between two cliffs drawn in fine vertical lines