Lune

EMNLP2024Top-tier venue

GRIZAL: Generative Prior-guided Zero-Shot Temporal Action Localization

Onkar Susladkar, Gayatri Deshmukh, Vandan Gorade, Sparsh Mittal

Abstract

Zero-shot temporal action localization (TAL) aims to temporally localize actions in videos without prior training examples. To address the challenges of TAL, we offer GRIZAL, a model that uses multimodal embeddings and dynamic motion cues to localize actions effectively. GRIZAL achieves sample diversity by using large-scale generative models such as GPT-4 for generating textual augmentations and DALL-E for generating image augmentations. Our model integrates vision-language embeddings with optical flow insights, optimized through a blend of supervised and self-supervised loss functions. On Activi-tyNet, Thumos14 and Charades-STA datasets, GRIZAL vastly outperforms state-of-the-art zero-shot TAL models, demonstrating its robustness and adaptability across a wide range of video content. The code and models are available on https://github.com/CandleLabAI/ GRIZAL-EMNLP2024 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Zero-Shot Text-to-Image Generation
  • Zero-Shot Text-to-Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion cites High-Resolution Image Synthesis with Latent Diffusion Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete DiffusionOnkar Kishor Susladkar, Jishu Sen Gupta, Chirag Sehgal, Sparsh Mittal et al.ICLR 2025

Builds on22

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
  4. BMN: Boundary-Matching Network for Temporal Action Proposal GenerationTianwei Lin, Xiao Liu, Xin Li, Errui Ding et al.ICCV 2019 · 709 citations
  5. Learning 2D Temporal Adjacent Networks for Moment Localization with Natural LanguageSongyang Zhang, Houwen Peng, Jianlong Fu, Jiebo LuoAAAI 2020 · 579 citations

Related papers

  1. Test-Time Zero-Shot Temporal Action LocalizationBenedetta Liberatori, Alessandro Conti, Paolo Rota, Yiming Wang et al.CVPR 2024
  2. Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video LocalizationXiongwen Deng, Haoyu Tang, Han Jiang, Qinghai Zheng et al.AAAI 2025
  3. Zero-shot Natural Language Video LocalizationJinwoo Nam, Daechul Ahn, Dongyeop Kang, Seong Jong Ha et al.ICCV 2021 · 60 citations
  4. Learning Generalized Representations for Open-Set Temporal Action LocalizationJunshan Hu, Liansheng Zhuang, Weisong Dong, Shiming Ge et al.ACM MM 2023 · 4 citations
  5. Memory Matters: Boosting Training-Free Zero-Shot Temporal Action Localization with a Learnable Lookup TableHan Jiang, Haoyu Tang, Xiaoxuan Mu, Chen Li et al.CVPR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines