Accelerating Long Video Understanding via Compressed Scene Graph-Enabled Chain-of-Thought

Long video understanding, which leverages Video-LLMs to analyze and interpret extended video content to extract meaningful information, insights, or summaries, is a fundamental task in multimedia domain. Chain-of-thought (CoT) methods are widely adopted to enhance long video understanding by incorporating intermediate reasoning steps. However, the iterative nature of CoT-requiring a lengthy sequence of internal thoughts-significantly increases the latency of video object description. To address this challenge, we design Compressed Scene Graph-enabled CoT (CSGCoT), a novel approach that facilitates efficient and accurate long-video object description. Inspired by video codec principles, we propose a compressed scene graph composed of two components: Key-SG for key frames and Delta-SG for delta frames, enabling efficient encoding of scene information across video segments. Specifically, CSGCoT comprises three major modules: (1) a Key-SG Detector that identifies representative segments, (2) a Delta-SG Generator that produces compensated representations for delta segments, and (3) a SG-Query Manager that converts scene graphs into natural language prompts for video object description. Experiments show that CSGCoT achieves comparable accuracy to SOTA methods while reducing latency by over 62.6% on hour-long videos while maintaining comparable accuracy.