EMNLP Findings2024
A Robust Dual-debiasing VQA Model based on Counterfactual Causal Effect
Lingyun Song, Chengkun Yang, Xuanyu Li, Xuequn Shang
Abstract
Traditional VQA models are inherently vulnerable to language bias, resulting in a significant performance drop when encountering out-of-distribution datasets. The conventional VQA models suffer from language bias that indicates a spurious correlation between textual questions and answers. Given the outstanding effectiveness of counterfactual causal inference in eliminating bias, we propose a modelagnostic dual-debiasing framework based on Counterfactual Causal Effect (DCCE), which explicitly models two types of language bias (i.e., shortcut and distribution bias) by separate branches under the counterfactual inference framework. The effects of both types of bias on answer prediction can be effectively mitigated by subtracting direct effect of textual questions on answers from total effect of visual questions on answers. Experimental results demonstrate that our proposed DCCE framework significantly reduces language bias and achieves state-of-the-art performance on the benchmark datasets without requiring additional augmented data. Our code is available in https://github.com/sxycyck/dcce.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Counterfactual Samples Synthesizing for Robust Visual Question Answering cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
- Greedy Gradient Ensemble for Robust Visual Question Answering cites Counterfactual Samples Synthesizing for Robust Visual Question Answering
- Greedy Gradient Ensemble for Robust Visual Question Answering cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
- Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA cites MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
- Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA cites Counterfactual Samples Synthesizing for Robust Visual Question Answering
- Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Cited by top-tier papers2
Builds on8
- Taking a HINT: Leveraging Explanations to Make Vision and Language Models More GroundedRamprasaath Ramasamy Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin et al.ICCV 2019 · 288 citations
- MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question AnsweringTejas Gokhale, Pratyay Banerjee, Chitta Baral, Yezhou YangEMNLP 2020 · 136 citations
- Greedy Gradient Ensemble for Robust Visual Question AnsweringXinzhe Han, Shuhui Wang, Chi Su, Qingming Huang et al.ICCV 2021 · 94 citations
- Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAQingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin et al.EMNLP Findings 2022 · 17 citations
- Counterfactual Samples Synthesizing for Robust Visual Question AnsweringLong Chen, Xin Yan, Jun Xiao, Hanwang Zhang et al.CVPR 2020
Related papers
- Counterfactual VQA: A Cause-Effect Look at Language BiasYulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu et al.CVPR 2021
- A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question AnsweringYu Li, Bojie Hu, Fengshuo Zhang, Yahan Yu et al.ACL Findings 2023 · 11 citations
- Language-Bias-Resilient Visual Question Answering via Adaptive Multi-Margin Collaborative DebiasingHuanjia Zhu, Shuyuan Zheng, Yishu Liu, Sudong Cai et al.NeurIPS 2025 · 2 citations
- COCA: COllaborative CAusal Regularization for Audio-Visual Question AnsweringMingrui Lao, Nan Pu, Yu Liu, Kai He et al.AAAI 2023 · 28 citations
- Counterfactual Vision and Language LearningEhsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Shi et al.CVPR 2020
