EMNLP Findings2023
Measuring Faithful and Plausible Visual Grounding in VQA
Abstract
Metrics for Visual Grounding (VG) in Visual Question Answering (VQA) systems primarily aim to measure a system's reliance on relevant parts of the image when inferring an answer to the given question. Lack of VG has been a common problem among state-of-the-art VQA systems and can manifest in over-reliance on irrelevant image parts or a disregard for the visual modality entirely. Although inference capabilities of VQA models are often illustrated by a few qualitative illustrations, most systems are not quantitatively assessed for their VG properties. We believe, an easily calculated criterion for meaningfully measuring a system's VG can help remedy this shortcoming, as well as add another valuable dimension to model evaluations and analysis. To this end, we propose a new VG metric that captures if a model a) identifies question-relevant objects in the scene, and b) actually relies on the information contained in the relevant objects when producing its answer, i.e., if its visual grounding is both "faithful" and "plausible". Our metric, called Faithful & Plausible Visual Grounding (FPVG), is straightforward to determine for most VQA model designs.
We give a detailed description of FPVG and evaluate several reference systems spanning various VQA architectures. Code to support the metric calculations on the GQA data set is available on GitHub 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answering cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
- Greedy Gradient Ensemble for Robust Visual Question Answering cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
- Uncovering the Full Potential of Visual Grounding Methods in VQA cites SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answering
- Uncovering the Full Potential of Visual Grounding Methods in VQA cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
- Seeing is Solving: Unlocking Efficient Multimodal RL via View Alignment cites Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Cited by top-tier papers2
Builds on12
- Taking a HINT: Leveraging Explanations to Make Vision and Language Models More GroundedRamprasaath Ramasamy Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin et al.ICCV 2019 · 288 citations
- Greedy Gradient Ensemble for Robust Visual Question AnsweringXinzhe Han, Shuhui Wang, Chi Su, Qingming Huang et al.ICCV 2021 · 94 citations
- Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang et al.ICML 2020 · 74 citations
- SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question AnsweringVipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang et al.CVPR 2022 · 41 citations
- ERASER: A Benchmark to Evaluate Rationalized NLP ModelsJay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric P. Lehman et al.ACL 2020 · 36 citations
Related papers
- DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and GroundingShilong Liu, Shijia Huang, Feng Li, Hao Zhang et al.AAAI 2023 · 44 citations
- Acknowledging Focus Ambiguity in Visual QuestionsChongyan Chen, Yu-Yun Tseng, Zhuoheng Li, Anush Venkatesh et al.ICCV 2025 · 1 citation
- Roses Are Red, Violets Are Blue... but Should VQA Expect Them To?Corentin Kervadec, Grigory Antipov, Moez Baccouche, Christian WolfCVPR 2021
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question AnsweringYushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang et al.ICCV 2023 · 400 citations
- Invariant Grounding for Video Question AnsweringYicong Li, Xiang Wang, Junbin Xiao, Wei Ji et al.CVPR 2022 · 108 citations
