Lune

ACL Findings2025

VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation

Xinlong Chen, Yuanxing Zhang, Chongling Rao, Yushuo Guan, Jiaheng Liu, Fuzheng Zhang, Chengru Song, Qiang Liu, Di Zhang, Tieniu Tan

Abstract

The training of controllable text-to-video (T2V) models relies heavily on the alignment between videos and captions, yet little existing research connects video caption evaluation with T2V generation assessment. This paper introduces VidCapBench, a video caption evaluation scheme specifically designed for T2V generation, agnostic to any particular caption format. VidCapBench employs a data annotation pipeline, combining expert model labeling and human refinement, to associate each collected video with key information spanning video aesthetics, content, motion, and physical laws. VidCapBench then partitions these key information attributes into automatically assessable and manually assessable subsets, catering to both the rapid evaluation needs of agile development and the accuracy requirements of thorough validation. By evaluating numerous state-of-the-art captioning models, we demonstrate the superior stability and comprehensiveness of VidCapBench compared to existing video captioning evaluation approaches. Verification with off-the-shelf T2V models reveals a significant positive correlation between scores on VidCapBench and the T2V quality evaluation metrics, indicating that VidCapBench can provide valuable guidance for training T2V models. The project is available at https: //github.com/VidCapBench/VidCapBench.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Scalable Diffusion Models with Transformers cites Learning Transferable Visual Models From Natural Language Supervision
  • Dense Text-to-Image Generation with Attention Modulation cites Learning Transferable Visual Models From Natural Language Supervision
  • AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration cites OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Reward
  • Time Blindness: Why Video-Language Models Can't See What Humans Can? cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites Scalable Diffusion Models with Transformers
  • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. AVoCaDO: An Audiovisual Video Captioner Driven by Temporal OrchestrationXinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua et al.ICLR 2026 · 27 citations
  2. Time Blindness: Why Video-Language Models Can't See What Humans Can?Ujjwal Upadhyay, Mukul Ranjan, Zhiqiang Shen, Mohamed ElhoseinyCVPR 2026 · 17 citations
  3. IF-VidCap: Can Video Caption Models Follow Instructions?Shihao Li, Yuanxing Zhang, Jiangtao Wu, Zhide Lei et al.ICLR 2026 · 7 citations
  4. VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy TasksXinlong Chen, Yuanxing Zhang, Yushuo Guan, Weihong Lin et al.ICLR 2026 · 7 citations
  5. OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence RewardChunlin Zhong, Qiuxia Hou, Zhangjun Zhou, Yanhao Zhang et al.AAAI 2026 · 4 citations

Builds on21

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
  3. Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari et al.ICML 2024 · 3,620 citations
  4. VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language ResearchXin Wang, Jiawei Wu, Jun-Kun Chen, Lei Li et al.ICCV 2019 · 688 citations
  5. Dense Text-to-Image Generation with Attention ModulationYunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha et al.ICCV 2023 · 204 citations

Related papers

  1. VC4VG: Optimizing Video Captions for Text-to-Video GenerationYang Du, Zhuoran Lin, Kaiqiang Song, Biao Wang et al.EMNLP 2025
  2. VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality EvaluationShi-Xue Zhang, Hongfa Wang, Duojun Huang, Xin Li et al.AAAI 2026 · 5 citations
  3. AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video GenerationZiwei Zhou, Zeyuan Lai, Rui Wang, Yifan Yang et al.ICML 2026 · 8 citations
  4. Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question AnsweringZizhen Wang, Bo Feng, Zhengfeng Lai, Shiyu Li et al.ACL 2026
  5. VBench: Comprehensive Benchmark Suite for Video Generative ModelsZiqi Huang, Yinan He, Jiashuo Yu, Fan Zhang et al.CVPR 2024
Dusk over the sea between two cliffs drawn in fine vertical lines