Lune

EMNLP Findings2024

A Robust Dual-debiasing VQA Model based on Counterfactual Causal Effect

Lingyun Song, Chengkun Yang, Xuanyu Li, Xuequn Shang

Abstract

Traditional VQA models are inherently vulnerable to language bias, resulting in a significant performance drop when encountering out-of-distribution datasets. The conventional VQA models suffer from language bias that indicates a spurious correlation between textual questions and answers. Given the outstanding effectiveness of counterfactual causal inference in eliminating bias, we propose a modelagnostic dual-debiasing framework based on Counterfactual Causal Effect (DCCE), which explicitly models two types of language bias (i.e., shortcut and distribution bias) by separate branches under the counterfactual inference framework. The effects of both types of bias on answer prediction can be effectively mitigated by subtracting direct effect of textual questions on answers from total effect of visual questions on answers. Experimental results demonstrate that our proposed DCCE framework significantly reduces language bias and achieves state-of-the-art performance on the benchmark datasets without requiring additional augmented data. Our code is available in https://github.com/sxycyck/dcce.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Counterfactual Samples Synthesizing for Robust Visual Question Answering cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
  • Greedy Gradient Ensemble for Robust Visual Question Answering cites Counterfactual Samples Synthesizing for Robust Visual Question Answering
  • Greedy Gradient Ensemble for Robust Visual Question Answering cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
  • Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA cites MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
  • Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA cites Counterfactual Samples Synthesizing for Robust Visual Question Answering
  • Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQAMin Hyuk Kim, Changheon Kim, Seok Bong YooEMNLP 2025 · 2 citations
  2. Counterfactual-Driven Zero-Shot Classifier ExpansionXiangyu Wang, Yanze Gao, Changxin Rong, Lyuzhou Chen et al.AAAI 2026 · 1 citation

Builds on8

  1. Taking a HINT: Leveraging Explanations to Make Vision and Language Models More GroundedRamprasaath Ramasamy Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin et al.ICCV 2019 · 288 citations
  2. MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question AnsweringTejas Gokhale, Pratyay Banerjee, Chitta Baral, Yezhou YangEMNLP 2020 · 136 citations
  3. Greedy Gradient Ensemble for Robust Visual Question AnsweringXinzhe Han, Shuhui Wang, Chi Su, Qingming Huang et al.ICCV 2021 · 94 citations
  4. Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAQingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin et al.EMNLP Findings 2022 · 17 citations
  5. Counterfactual Samples Synthesizing for Robust Visual Question AnsweringLong Chen, Xin Yan, Jun Xiao, Hanwang Zhang et al.CVPR 2020

Related papers

  1. Counterfactual VQA: A Cause-Effect Look at Language BiasYulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu et al.CVPR 2021
  2. A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question AnsweringYu Li, Bojie Hu, Fengshuo Zhang, Yahan Yu et al.ACL Findings 2023 · 11 citations
  3. Language-Bias-Resilient Visual Question Answering via Adaptive Multi-Margin Collaborative DebiasingHuanjia Zhu, Shuyuan Zheng, Yishu Liu, Sudong Cai et al.NeurIPS 2025 · 2 citations
  4. COCA: COllaborative CAusal Regularization for Audio-Visual Question AnsweringMingrui Lao, Nan Pu, Yu Liu, Kai He et al.AAAI 2023 · 28 citations
  5. Counterfactual Vision and Language LearningEhsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Shi et al.CVPR 2020
Dusk over the sea between two cliffs drawn in fine vertical lines