EMNLP Findings2025
QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering
Woojun Jung, Junyeong Kim
Abstract
Video-to-text summarization remains underexplored in terms of comprehensive evaluation methods. Traditional n-gram overlapbased metrics and recent large language model (LLM)-based approaches depend heavily on human-written reference summaries, limiting their practicality and sensitivity to nuanced semantic aspects. In this paper, we propose QEVA, a reference-free metric evaluating candidate summaries directly against source videos through multimodal question answering. QEVA assesses summaries along three clear dimensions: Coverage, Factuality, and Chronology. We also introduce MLVU(VS)-Eval, a new annotated benchmark derived from the MLVU dataset, comprising 800 summaries generated from 200 videos using state-of-theart video-language multimodal models. This dataset establishes a transparent and consistent framework for evaluation. Experimental results demonstrate that QEVA shows higher correlation with human judgments compared to existing approaches, as measured by Kendall's τ b , τ c , and Spearman's ρ. We hope that our benchmark and metric will facilitate meaningful progress in video-to-text summarization research and provide valuable insights for the development of future evaluation methods. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Asking and Answering Questions to Evaluate the Factual Consistency of Summaries cites BERTScore: Evaluating Text Generation with BERT
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering cites Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering cites CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering cites BERTScore: Evaluating Text Generation with BERT
- Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering cites CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning cites BERTScore: Evaluating Text Generation with BERT
Cited by top-tier papers1
Builds on9
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- CLIPScore: A Reference-free Evaluation Metric for Image CaptioningJack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras et al.EMNLP 2021 · 937 citations
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic AlignmentBin Zhu, Bin Lin, Munan Ning, Yang Yan et al.ICLR 2024 · 403 citations
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question AnsweringYushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang et al.ICCV 2023 · 400 citations
- Asking and Answering Questions to Evaluate the Factual Consistency of SummariesAlex Wang, Kyunghyun Cho, Mike LewisACL 2020 · 317 citations
Related papers
- Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and DiversityAbid Ali, Diego Mollá, Usman NaseemACL Findings 2026
- MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering BenchmarkShaden Shaar, Bradon Thymes, Sirawut Chaixanien, Claire Cardie et al.CVPR 2026 · 4 citations
- VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality EvaluationShi-Xue Zhang, Hongfa Wang, Duojun Huang, Xin Li et al.AAAI 2026 · 5 citations
- VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual AnalysisShubhashis Roy Dipta, Tz-Ying Wu, Subarna TripathiACL 2026
- One Prompt To Rule Them All: LLMs for Opinion Summary EvaluationTejpalsingh Siledar, Swaroop Nath, Sankara Sri Raghava Ravindra Muddu, Rupasai Rangaraju et al.ACL 2024
