TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding. However, how well do the models truly perform visual temporal reasoning? Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame Information Disparity. Following these principles, we introduce TOMATO, TempOral Reasoning MultimodAl EvaluaTiOn, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding. TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks (i.e., action count, direction, rotation, shape & trend, velocity & frequency, and visual cues), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centric, real-world, and simulated scenarios. Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model. Moreover, our in-depth analysis uncovers more fundamental limitations beyond this gap in current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as a continuous sequence. We believe TOMATO will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI systems capable of comprehending the human world dynamics through the video modality. * Equal Contribution. TOMATO (Ours) TempCompass (Liu et al., 2024): What are the woman athletes doing? A. Cycling B. Swimming C. Running D. Dancing VITATECS (Li et al., 2023): Which of the following best describes the content of the video? A. Cheese is being spread B. Cheese is being sliced MVBench (Li et al., 2024): What is behind Monica when she is in the chair talking to Ross? A. A television B. Joey's white dog sculpture C. An American flag D. A Christmas tree E. A basket of laundry ReXTime (Chen et al., 2024): Why is the woman preparing ingredients and utensils? A.To make herself a sandwich B. To cook dinner for her family C. To bake a cake D. To prepare a salad In which direction(s) does the Ping Pong ball rotate? A. Clockwise throughout B. Counter-clockwise then clockwise C. Counter-clockwise throughout D. Clockwise then counter-clockwise