Discovering Spatio-Temporal Rationales for Video Question Answering

This paper strives to solve complex video question answering (VideoQA) which features long video containing multiple objects and events at different time. To tackle the challenge, we highlight the importance of identifying question-critical temporal moments and spatial objects from the vast amount of video content. Towards this, we propose a Spatio-Temporal Rationalization (STR), a differentiable selection module that adaptively collects questioncritical moments and objects using cross-modal interaction. The discovered video moments and objects are then served as grounded rationales to support answer reasoning. Based on STR, we further propose TranSTR, a Transformerstyle neural network architecture that takes STR as the core and additionally underscores a novel answer interaction mechanism to coordinate STR for answer decoding. Experiments on four datasets show that TranSTR achieves new state-of-the-art (SoTA). Especially, on NExT-QA and Causal-VidQA which feature complex VideoQA, it significantly surpasses the previous SoTA by 5.8% and 6.8%, respectively. We then conduct extensive studies to verify the importance of STR as well as the proposed answer interaction mechanism. With the success of TranSTR and our comprehensive analysis, we hope this work can spark more future efforts in complex VideoQA. Code will be released at https://github.com/yl3800/TranSTR . Answ A rid B loo C pu D rep E tur Question: What does the second person do after stopping her bike? Answer Candidates: A ride the bike B look at the girl C push the bike D repair road E turn around Answer: push the bike stopping Top Acc (%) 100% 70% 40% 10% Top % Video Length Short Long 272649 Acc (%) 61 +0.7 -1.5 -2.0 -1.3 (a) A example of long video (52s) with multiple objects, the question-related frames and objects are located to support the reasoning. Que seco stop Answer Candida A ride the bike B look at the gir C push the bike D repair road E turn around Question: What does the second person do after stopping her bike? Answer Candidates: A ride the bike B look at the girl C push the bike D repair road E turn around Answer: push the bike stopping Top percentag Acc (%) 100% 70% 40% 10% Top Percentage by Video Length 2726497009 Acc (%) 61 +0.7 -1.5 -2.0 -1.3 (b) Accuracy by video length. Q se st Answer Cand A ride the bik B look at the C push the bik D repair road E turn around Question: What does the second person do after stopping her bike? Answer Candidates: A ride the bike B look at the girl C push the bike D repair road E turn around Answer: push the bike stopping Top percent Acc (%) 100% 70% 40% 10% Top % Video Length Short Long