GRIZAL: Generative Prior-guided Zero-Shot Temporal Action Localization
Onkar Susladkar, Gayatri Deshmukh, Vandan Gorade, Sparsh Mittal
Abstract
Zero-shot temporal action localization (TAL) aims to temporally localize actions in videos without prior training examples. To address the challenges of TAL, we offer GRIZAL, a model that uses multimodal embeddings and dynamic motion cues to localize actions effectively. GRIZAL achieves sample diversity by using large-scale generative models such as GPT-4 for generating textual augmentations and DALL-E for generating image augmentations. Our model integrates vision-language embeddings with optical flow insights, optimized through a blend of supervised and self-supervised loss functions. On Activi-tyNet, Thumos14 and Charades-STA datasets, GRIZAL vastly outperforms state-of-the-art zero-shot TAL models, demonstrating its robustness and adaptability across a wide range of video content. The code and models are available on https://github.com/CandleLabAI/ GRIZAL-EMNLP2024 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- High-Resolution Image Synthesis with Latent Diffusion Models cites Zero-Shot Text-to-Image Generation
- Zero-Shot Text-to-Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion cites High-Resolution Image Synthesis with Latent Diffusion Models
Cited by top-tier papers1
Builds on22
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
- BMN: Boundary-Matching Network for Temporal Action Proposal GenerationTianwei Lin, Xiao Liu, Xin Li, Errui Ding et al.ICCV 2019 · 709 citations
- Learning 2D Temporal Adjacent Networks for Moment Localization with Natural LanguageSongyang Zhang, Houwen Peng, Jianlong Fu, Jiebo LuoAAAI 2020 · 579 citations
Related papers
- Test-Time Zero-Shot Temporal Action LocalizationBenedetta Liberatori, Alessandro Conti, Paolo Rota, Yiming Wang et al.CVPR 2024
- Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video LocalizationXiongwen Deng, Haoyu Tang, Han Jiang, Qinghai Zheng et al.AAAI 2025
- Zero-shot Natural Language Video LocalizationJinwoo Nam, Daechul Ahn, Dongyeop Kang, Seong Jong Ha et al.ICCV 2021 · 60 citations
- Learning Generalized Representations for Open-Set Temporal Action LocalizationJunshan Hu, Liansheng Zhuang, Weisong Dong, Shiming Ge et al.ACM MM 2023 · 4 citations
- Memory Matters: Boosting Training-Free Zero-Shot Temporal Action Localization with a Learnable Lookup TableHan Jiang, Haoyu Tang, Xiaoxuan Mu, Chen Li et al.CVPR 2026
