ACL Findings2025
Generative Frame Sampler for Long Video Understanding
Linli Yao, Haoning Wu, Kun Ouyang, Yuanxing Zhang, Caiming Xiong, Bei Chen, Xu Sun, Junnan Li
Abstract
Despite recent advances in Video Large Language Models (VideoLLMs), effectively understanding long-form videos remains a significant challenge. Perceiving lengthy videos containing thousands of frames poses substantial computational burden. To mitigate this issue, this paper introduces Generative Frame Sampler (GenS), a plug-and-play module integrated with VideoLLMs to facilitate efficient lengthy video perception. Built upon a lightweight VideoLLM, GenS leverages its inherent vision-language capabilities to identify question-relevant frames. To facilitate effective retrieval, we construct GenS-Video-150K, a large-scale video instruction dataset with dense frame relevance annotations. Extensive experiments demonstrate that GenS consistently boosts the performance of various Vide-oLLMs, including open-source models (Qwen2-VL-7B, Aria-25B, VILA-40B, LLaVA-Video-7B/72B) and proprietary assistants (GPT-4o, Gemini). When equipped with GenS, opensource VideoLLMs achieve impressive stateof-the-art results on long-form video benchmarks: LLaVA-Video-72B reaches 66.8 (+4.3) on LongVideoBench and 77.0 (+2.7) on MLVU, while Aria obtains 39.2 on HourVideo surpassing the Gemini-1.5-pro by 1.9 points. We release the code, dataset and models at https: //generative-sampler.github.io.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
16 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence cites Sigmoid Loss for Language Image Pre-Training
- Sigmoid Loss for Language Image Pre-Training cites Learning Transferable Visual Models From Natural Language Supervision
- VideoITG: Multimodal Video Understanding with Instructed Temporal Grounding cites Sigmoid Loss for Language Image Pre-Training
- VideoITG: Multimodal Video Understanding with Instructed Temporal Grounding cites Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models cites Learning Transferable Visual Models From Natural Language Supervision
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs cites Sigmoid Loss for Language Image Pre-Training
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers16
- TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMsJun Zhang, Teng Wang, Yuying Ge, Yixiao Ge et al.CVPR 2026 · 48 citations
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement LearningLong Xing, Xiaoyi Dong, Yuhang Zang, Yuhang Cao et al.ICLR 2026 · 37 citations
- VideoITG: Multimodal Video Understanding with Instructed Temporal GroundingShihao Wang, Guo Chen, De-An Huang, Zhiqi Li et al.CVPR 2026 · 35 citations
- VideoLucy: Deep Memory Backtracking for Long Video UnderstandingJialong Zuo, Yongtai Deng, Lingdong Kong, Jingkang Yang et al.NeurIPS 2025 · 23 citations
- Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual EvidenceKun Ouyang, Yuanxin Liu, Linli Yao, Yishuo Cai et al.CVPR 2026 · 17 citations
Builds on24
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMsPeter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo et al.NeurIPS 2024 · 1,004 citations
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language ModelsMuhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Fahad KhanACL 2024 · 279 citations
Related papers
- MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video UnderstandingWenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen et al.CVPR 2026 · 6 citations
- Unleashing Hour-Scale Video Training for Long Video-Language UnderstandingJingyang Lin, Jialian Wu, Ximeng Sun, Ze Wang et al.NeurIPS 2025 · 25 citations
- A Training-Free Framework for Long Video Understanding via Video-Query-Options SimilarityZhirong Wu, Xiaodong Wang, Langling Huang, Teng Xu et al.ICLR 2026
- IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMsDavid Ma, Yuanxing Zhang, Jincheng Ren, Jiawei Guo et al.ICLR 2026 · 5 citations
- M-LLM Based Video Frame Selection for Efficient Video UnderstandingKai Hu, Feng Gao, Xiaohan Nie, Peng Zhou et al.CVPR 2025
