LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer

Current approaches to Video Question Answering (VideoQA) primarily focus on cross-modality matching, which is limited by the requirement for extensive data annotations and the insufficient capacity for causal reasoning (e.g. attributing accidents). To address these challenges, we introduce a causal framework for video reasoning, termed Learning Latent Causal Processes (LLCP). At the heart of LLCP lies a multivariate generative model designed to analyze the spatial-temporal dynamics of objects within events. Leveraging the inherent modularity of causal mechanisms, we train the model through self-supervised local auto-regression eliminating the need for annotated question-answer pairs. During inference, the model is applied to answer two types of reasoning questions: accident attribution, which infers the cause from observed effects, and counterfactual prediction, which predicts the effects of counterfactual conditions given the factual evidence. In the first scenario, we identify variables that deviate from the established distribution by the learned model, signifying the root cause of accidents. In the second, we replace embeddings of previous variables with counterfactual ones, enabling us to forecast potential developments. Once we have identified these cause/effect variables, natural language answers are derived through a combination of grammatical parsing and a pre-trained vision-language model. We assess the efficacy of LLCP on both synthetic and real-world data, demonstrating comparable performance to supervised methods despite our framework using no paired textual annotations. The code is available at https://github.com/CHENGY12/LLCP . * Equal contribution. METHOD In this section, we first introduce the TMGM model and then describe how we can leverage this model in the inference for both cause-based and effect-based video reasoning tasks. TEMPORAL MULTIVARIATE GENERATIVE MODEL Causal Processes. Without loss of generality, we assume the given video X consists of T frames and N agents as X = x t,i | T,N t=1,i=1 . For each variable, we introduce our data-generating pro- Keywords Vehicles and Distance