CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

Humans are able to perceive, understand and reason about causal events. Developing models with similar physical and causal understanding capabilities is a long-standing goal of artificial intelligence. As a step towards this direction, we introduce CRAFT 1 , a new video question answering dataset that requires causal reasoning about physical forces and object interactions. It contains 58K video and question pairs that are generated from 10K videos from 20 different virtual environments, containing various objects in motion that interact with each other and the scene. Two question categories in CRAFT include previously studied descriptive and counterfactual questions. Additionally, inspired by the Force Dynamics Theory in cognitive linguistics, we introduce a new causal question category that involves understanding the causal interactions between objects through notions like cause, enable, and prevent. Our results show that even though the questions in CRAFT are easy for humans, the tested baseline models, including existing state-of-the-art methods, do not yet deal with the challenges posed in our benchmark. * indicates equal contributions. 1 Data and code available on our project website at https://sites.google.com/view/craft-benchmark Descriptive Questions Counterfactual Questions Cause Enable Prevent Q: "How many objects fall to the ground?" A: "2" Q: "After entering the basket, does the small yellow square collide with other objects?" A: "True" Q: "Will the small gray box enter the basket if any of the other objects are removed?" A: "True" Q: "How many objects fall to the ground if the small yellow box is removed?" A: "1" Q: "Does the small brown sphere cause the tiny yellow box to enter the basket?" A: "True" Q: "Does the small brown sphere enable the small yellow box to enter the basket?" A: "False" Q: "Does the small brown ball cause the big gray triangle to fall to the ground?" A: "False" Q: "How many objects does the small gray block enable to enter the basket?" A: "0" Q: "Does the small yellow square prevent the tiny brown circle from entering the basket?" A: "True" Q: "How many objects does the large cyan triangle prevent from entering the basket?" A: "1" Ground Causal Questions