EMNLP Findings2024
Exploring Question Guidance and Answer Calibration for Visually Grounded Video Question Answering
Yuanxing Xu, Yuting Wei, Shuai Zhong, Xinming Chen, Jinsheng Qi, Bin Wu
Abstract
Video Question Answering (VideoQA) tasks require not only correct answers but also visual evidence. The "localize-then-answer" strategy, while enhancing accuracy and interpretability, faces challenges due to the lack of temporal localization labels in VideoQA datasets. Existing methods often train the models' localization capabilities indirectly using QA labels, leading to inaccurate localization. Moreover, our experiments show that despite high accuracy, current models depend too heavily on language shortcuts or spurious correlations with irrelevant visual context. To address these issues, we propose a Question-Guided and Answer-Calibrated TRansformer (QGAC-TR), which guides and calibrates localization using question and option texts without localization labels. Furthermore, we design two self-supervised learning tasks to further enhance the model's refined localization capabilities. Extensive experiments on three public datasets focused on temporal and causal reasoning show that our model not only achieves accuracy comparable to large-scale pretrained models but also leads in localization aspects. Code will be released at this link.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Question-Answering Dense Video Events cites Learning Transferable Visual Models From Natural Language Supervision
- Question-Answering Dense Video Events cites Flamingo: a Visual Language Model for Few-Shot Learning
- Question-Answering Dense Video Events cites Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models cites Learning Transferable Visual Models From Natural Language Supervision
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models cites Reasoning with Heterogeneous Graph Alignment for Video Question Answering
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
Cited by top-tier papers1
Builds on22
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- Learning 2D Temporal Adjacent Networks for Moment Localization with Natural LanguageSongyang Zhang, Houwen Peng, Jianlong Fu, Jiebo LuoAAAI 2020 · 579 citations
- Zero-Shot Video Question Answering via Frozen Bidirectional Language ModelsAntoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev et al.NeurIPS 2022 · 305 citations
- Reasoning with Heterogeneous Graph Alignment for Video Question AnsweringPin Jiang, Yahong HanAAAI 2020 · 214 citations
Related papers
- Can I Trust Your Answer? Visually Grounded Video Question AnsweringJunbin Xiao, Angela Yao, Yicong Li, Tat-Seng ChuaCVPR 2024 · 44 citations
- Invariant Grounding for Video Question AnsweringYicong Li, Xiang Wang, Junbin Xiao, Wei Ji et al.CVPR 2022 · 108 citations
- Visual Causal Scene Refinement for Video Question AnsweringYushen Wei, Yang Liu, Hong Yan, Guanbin Li et al.ACM MM 2023 · 31 citations
- Equivariant and Invariant Grounding for Video Question AnsweringYicong Li, Xiang Wang, Junbin Xiao, Tat-Seng ChuaACM MM 2022 · 33 citations
- Language-Guided Visual Aggregation Network for Video Question AnsweringXiao Liang, Di Wang, Quan Wang, Bo Wan et al.ACM MM 2023 · 5 citations
