Lune

ICLR2020Top-tier venue

CLEVRER: Collision Events for Video Representation and Reasoning

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum

Abstract

The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset, a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human causal judgment, CLEVRER includes four types of question: descriptive (e.g., 'what color'), explanatory ('what's responsible for'), predictive ('what will happen next'), and counterfactual ('what if'). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perceptionbased task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

187 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CATER: A diagnostic dataset for Compositional Actions & TEmporal Reasoning cites CoPhy: Counterfactual Learning of Physical Dynamics
  • Conditional Object-Centric Learning from Video cites Object-Centric Learning with Slot Attention
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers187

  1. Object-Centric Learning with Slot AttentionFrancesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran et al.NeurIPS 2020 · 1,275 citations
  2. Conditional Object-Centric Learning from VideoThomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone et al.ICLR 2022 · 290 citations
  3. CATER: A diagnostic dataset for Compositional Actions & TEmporal ReasoningRohit Girdhar, Deva RamananICLR 2020 · 198 citations
  4. PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable PhysicsZhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou et al.ICLR 2021 · 164 citations
  5. Medical Visual Question Answering via Conditional ReasoningLi-Ming Zhan, Bo Liu, Lu Fan, Jiaxin Chen et al.ACM MM 2020 · 157 citations

Builds on1

  1. CoPhy: Counterfactual Learning of Physical DynamicsFabien Baradel, Natalia Neverova, Julien Mille, Greg Mori et al.ICLR 2020 · 105 citations

Related papers

  1. Grounding Physical Concepts of Objects and Events Through Dynamic Visual ReasoningZhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong et al.ICLR 2021 · 13 citations
  2. CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractionsTayfun Ates, Muhammed Samil Atesoglu, Cagatay Yigit, Ilker Kesen et al.ACL Findings 2022 · 9 citations
  3. From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-AnsweringJiangtong Li, Li Niu, Liqing ZhangCVPR 2022 · 48 citations
  4. Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMsMohamad Ballout, Okajevo Wilfred, Seyedalireza Yaghoubi, Nohayr Abdelmoneim et al.EMNLP Findings 2025
  5. Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question AnsweringXingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines