ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries
Wangyu Xue, Chen Qian, Jiayi Wu, Yang Zhou, Wentao Liu, Ju Ren, Siming Fan, Yaoxue Zhang
Abstract
Existing research on human-centric video understanding typically focuses on analyzing specific moments or entire videos. However, many applications require higher precision at the frame level. In this work, we propose a novel task, BestShot, which aims to locate highlight frames within human-centric videos through language queries. This task requires not only a deep semantic understanding of human actions but also precise temporal localization. To support this task, we introduce the BestShot Benchmark. The benchmark is meticulously constructed by combining human-annotated highlight frames, duration labels and detailed textual descriptions. These descriptions cover three critical elements: (1) Visual content; (2) Fine-grained actions; and (3) Human pose descriptions. Together, these elements provide the necessary precision to identify the exact highlight frames in videos.
To tackle this problem, we have collected two distinct datasets: (i) ShotGPT4o Dataset, which is algorithmically generated by GPT-4o and (ii) Image-SMPLText Dataset, which features large-scale and accurate per-frame pose descriptions using PoseScript and existing pose estimation datasets. Based on these datasets, we present a strong baseline model, ShotVL, fine-tuned from InternVL, specifically for BestShot. We highlight the impressive zero-shot capabilities of our model and offer comparative analyses with existing state-of-the-art (SOTA) models. ShotVL demonstrates a significant 64% improvement over InternVL on the BestShot Benchmark and a notable 68% improvement on the THUMOS14 Benchmark, while maintaining SOTA performance in general image classification and retrieval.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on11
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric RenderingWei Cheng, Ruixiang Chen, Siming Fan, Wanqi Yin et al.ICCV 2023 · 106 citations
- Capturing and Inferring Dense Full-Body Human-Scene ContactChun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin et al.CVPR 2022 · 106 citations
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video UnderstandingShuhuai Ren, Linli Yao, Shicheng Li, Xu Sun et al.CVPR 2024 · 83 citations
- SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and ModelingZhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu et al.ICCV 2023 · 73 citations
Related papers
- ShotBench: Expert-Level Cinematic Understanding in Vision-Language ModelsHongbo Liu, Jingwen He, Yi Jin, Dian Zheng et al.NeurIPS 2025 · 24 citations
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video UnderstandingZichen Liu, Kunlun Xu, Bing Su, Xu Zou et al.CVPR 2025
- VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal GroundingYongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng et al.AAAI 2025 · 27 citations
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video RecognitionTom Tongjia Chen, Hongshan Yu, Zhengeng Yang, Zechuan Li et al.CVPR 2024
- GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?Yiyang Zhou, Linjie Li, Shi Qiu, Zhengyuan Yang et al.EMNLP 2025 · 2 citations
