Decompose and Conquer: Compositional Reasoning for Zero-Shot Temporal Action Localization
Haoyu Tang, Tianyuan Liang, Han Jiang, Xuesong Liu, Qinghai Zheng, Yupeng Hu
Abstract
Current Zero-Shot Temporal Action Localization (ZSTAL) methods, whether training-based or training-free ones, still predominantly rely on a single, unified query to localize an entire action. This unified representation is fundamentally ill-suited for complex real-world activities, as it fails to capture their internal compositional structure and adapt to dynamic, multi-stage variations across videos. To address this, we regard ZSTAL as a compositional reasoning task and introduce CASCADE, a Context-Aware Staged Action DEcomposition framework. Inspired by the human cognitive process of perceiving context, decomposing events, and reconstructing instances, CASCADE follows a training-free pipeline. It first perceives the video's context by leveraging a Multimodal Large Language Model (MLLM) to both filter out irrelevant actions and then generate a rich, video-specific caption for each action present in the video. An LLM then decomposes this caption into multiple, temporally ordered stages, which serve as fine-grained queries to guide the MLLM in estimating frame-level confidence scores. Recognizing that this decomposition can fragment a single action, a novel hierarchical merging logic then reconstructs complete instances by intelligently fusing these preliminary temporal segments based on their semantic progression and coherence. Extensive experiments and ablation studies on THUMOS14 and ActivityNet-1.3 show that CASCADE not only sets a new state-of-the-art among training-free methods but, most notably, significantly outperforms all prior training-based approaches on ActivityNet-1.3.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Builds on18
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- GLIPv2: Unifying Localization and Vision-Language UnderstandingHaotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen et al.NeurIPS 2022 · 403 citations
- Interest-aware Message-Passing GCN for RecommendationFan Liu, Zhiyong Cheng, Lei Zhu, Zan Gao et al.WWW 2021 · 325 citations
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language ModelsMuhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Fahad KhanACL 2024 · 279 citations
- Video-LLaVA: Learning United Visual Representation by Alignment Before ProjectionBin Lin, Yang Ye, Bin Zhu, Jiaxi Cui et al.EMNLP 2024 · 231 citations
Related papers
- Decomposed Attention Fusion in MLLMs for Training-free Video Reasoning SegmentationSu Ho Han, Jeongseok Hyun, Pilhyeon Lee, Minho Shim et al.ICLR 2026 · 2 citations
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video GroundingZaiquan Yang, Yuhao Liu, Gerhard P. Hancke, Rynson W. H. LauNeurIPS 2025 · 10 citations
- RealVG: Unleashing MLLMs for Training-Free Spatio-Temporal Video Grounding in the WildHongchen Wei, Zhenzhong ChenACM MM 2025 · 1 citation
- CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for VideosShiu-Hong Kao, Yu-Wing Tai, Chi-Keung TangICLR 2026 · 8 citations
- Test-Time Zero-Shot Temporal Action LocalizationBenedetta Liberatori, Alessandro Conti, Paolo Rota, Yiming Wang et al.CVPR 2024
