GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?

Existing video benchmarks often resemble image-based benchmarks, with question types like "What actions does the person perform throughout the video?" or "What color is the woman's dress in the video?" For these, models can often answer by scanning just a few key frames, without deep temporal reasoning. This limits our ability to assess whether large visionlanguage models (LVLMs) can truly think with videos rather than perform superficial framelevel analysis. To address this, we introduce GLIMPSE, a benchmark specifically designed to evaluate whether LVLMs can genuinely think with videos. Unlike prior benchmarks, GLIMPSE emphasizes comprehensive video understanding beyond static image cues. It consists of 3,269 videos and over 4,342 highly visual-centric questions across 11 categories, including Trajectory Analysis, Temporal Reasoning, and Forensics Detection. All questions are carefully crafted by human annotators and require watching the entire video and reasoning over full video context-this is what we mean by thinking with video. These questions cannot be answered by scanning selected frames or relying on text alone. In human evaluations, GLIMPSE achieves 94.82% accuracy, but current LVLMs face significant challenges. Even the best-performing model, GPT-o3, reaches only 66.43%, highlighting that LVLMs still struggle to move beyond surface-level reasoning to truly think with videos. We publicly release our benchmark and code at https: //github.com/aiming-lab/GLIMPSE . Is the man who passes through the center of the screen at the beginning of the video closer to the position of the photographer when he passes through the center at the second time than he was at the start? A) The man is at the same distance ... B) The man is further away ... C) No, the man's position ... not changed. D) The man is closer to ...