Lune

ACL Findings2021

WeaQA: Weak Supervision via Captions for Visual Question Answering

Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, Chitta Baral

Abstract

Methodologies for training visual question answering (VQA) models assume the availability of datasets with human-annotated Image-Question-Answer (I-Q-A) triplets. This has led to heavy reliance on datasets and a lack of generalization to new types of questions and scenes. Linguistic priors along with biases and errors due to annotator subjectivity have been shown to percolate into VQA models trained on such samples. We study whether models can be trained without any human-annotated Q-A pairs, but only with images and their associated textual descriptions or captions. We present a method to train models with synthetic Q-A pairs generated procedurally from captions. Additionally, we demonstrate the efficacy of spatial-pyramid image patches as a simple but effective alternative to dense and costly object bounding box annotations used in existing VQA models. Our experiments on three VQA benchmarks demonstrate the efficacy of this weakly-supervised approach, especially on the VQA-CP challenge, which tests performance under changing linguistic priors.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

12 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Cited by top-tier papers12

  1. Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question AnsweringMan Luo, Yankai Zeng, Pratyay Banerjee, Chitta BaralEMNLP 2021 · 53 citations
  2. Improving Biomedical Information Retrieval with Neural RetrieversMan Luo, Arindam Mitra, Tejas Gokhale, Chitta BaralAAAI 2022 · 42 citations
  3. Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question PromptsYunshi Lan, Xiang Li, Xin Liu, Yang Li et al.ACM MM 2023 · 29 citations
  4. Exploring Question Decomposition for Zero-Shot VQAZaid Khan, Vijay Kumar B. G, Samuel Schulter, Manmohan Chandraker et al.NeurIPS 2023 · 26 citations
  5. Weakly Supervised Relative Spatial Reasoning for Visual Question AnsweringPratyay Banerjee, Tejas Gokhale, Yezhou Yang, Chitta BaralICCV 2021 · 19 citations

Builds on16

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  3. Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
  4. Meta-Learning to Detect Rare ObjectsYu-Xiong Wang, Deva Ramanan, Martial HebertICCV 2019 · 339 citations
  5. On the Value of Out-of-Distribution Testing: An Example of Goodhart's LawDamien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha et al.NeurIPS 2020 · 163 citations

Related papers

  1. Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic EditingVedika Agarwal, Rakshith Shetty, Mario FritzCVPR 2020
  2. Pseudo-Q: Generating Pseudo Language Queries for Visual GroundingHaojun Jiang, Yuanze Lin, Dongchen Han, Shiji Song et al.CVPR 2022 · 60 citations
  3. Linguistic Structures As Weak Supervision for Visual Scene Graph GenerationKeren Ye, Adriana KovashkaCVPR 2021
  4. VoQA: Visual-only Question AnsweringJianing An, Luyang Jiang, Jie Luo, Wenjun Wu, Lei HuangCVPR Findings 2026
  5. MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual CaptioningBang Yang, Fenglin Liu, Xian Wu, Yaowei Wang et al.ACL 2023 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines