CLEVRER: Collision Events for Video Representation and Reasoning
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum
Abstract
The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset, a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human causal judgment, CLEVRER includes four types of question: descriptive (e.g., 'what color'), explanatory ('what's responsible for'), predictive ('what will happen next'), and counterfactual ('what if'). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perceptionbased task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
187 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers187
- Object-Centric Learning with Slot AttentionFrancesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran et al.NeurIPS 2020 · 1,275 citations
- Conditional Object-Centric Learning from VideoThomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone et al.ICLR 2022 · 290 citations
- CATER: A diagnostic dataset for Compositional Actions & TEmporal ReasoningRohit Girdhar, Deva RamananICLR 2020 · 198 citations
- PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable PhysicsZhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou et al.ICLR 2021 · 164 citations
- Medical Visual Question Answering via Conditional ReasoningLi-Ming Zhan, Bo Liu, Lu Fan, Jiaxin Chen et al.ACM MM 2020 · 157 citations
Builds on1
Related papers
- Grounding Physical Concepts of Objects and Events Through Dynamic Visual ReasoningZhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong et al.ICLR 2021 · 13 citations
- CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractionsTayfun Ates, Muhammed Samil Atesoglu, Cagatay Yigit, Ilker Kesen et al.ACL Findings 2022 · 9 citations
- From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-AnsweringJiangtong Li, Li Niu, Liqing ZhangCVPR 2022 · 48 citations
- Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMsMohamad Ballout, Okajevo Wilfred, Seyedalireza Yaghoubi, Nohayr Abdelmoneim et al.EMNLP Findings 2025
- Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question AnsweringXingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen et al.ICLR 2025
