Lune

EMNLP Findings2025

QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering

Woojun Jung, Junyeong Kim

Abstract

Video-to-text summarization remains underexplored in terms of comprehensive evaluation methods. Traditional n-gram overlapbased metrics and recent large language model (LLM)-based approaches depend heavily on human-written reference summaries, limiting their practicality and sensitivity to nuanced semantic aspects. In this paper, we propose QEVA, a reference-free metric evaluating candidate summaries directly against source videos through multimodal question answering. QEVA assesses summaries along three clear dimensions: Coverage, Factuality, and Chronology. We also introduce MLVU(VS)-Eval, a new annotated benchmark derived from the MLVU dataset, comprising 800 summaries generated from 200 videos using state-of-theart video-language multimodal models. This dataset establishes a transparent and consistent framework for evaluation. Experimental results demonstrate that QEVA shows higher correlation with human judgments compared to existing approaches, as measured by Kendall's τ b , τ c , and Spearman's ρ. We hope that our benchmark and metric will facilitate meaningful progress in video-to-text summarization research and provide valuable insights for the development of future evaluation methods. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Asking and Answering Questions to Evaluate the Factual Consistency of Summaries cites BERTScore: Evaluating Text Generation with BERT
  • TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering cites Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
  • TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering cites CLIPScore: A Reference-free Evaluation Metric for Image Captioning
  • Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering cites BERTScore: Evaluating Text Generation with BERT
  • Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering cites CLIPScore: A Reference-free Evaluation Metric for Image Captioning
  • CLIPScore: A Reference-free Evaluation Metric for Image Captioning cites BERTScore: Evaluating Text Generation with BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question AnsweringZizhen Wang, Bo Feng, Zhengfeng Lai, Shiyu Li et al.ACL 2026

Builds on9

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. CLIPScore: A Reference-free Evaluation Metric for Image CaptioningJack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras et al.EMNLP 2021 · 937 citations
  3. LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic AlignmentBin Zhu, Bin Lin, Munan Ning, Yang Yan et al.ICLR 2024 · 403 citations
  4. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question AnsweringYushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang et al.ICCV 2023 · 400 citations
  5. Asking and Answering Questions to Evaluate the Factual Consistency of SummariesAlex Wang, Kyunghyun Cho, Mike LewisACL 2020 · 317 citations

Related papers

  1. Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and DiversityAbid Ali, Diego Mollá, Usman NaseemACL Findings 2026
  2. MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering BenchmarkShaden Shaar, Bradon Thymes, Sirawut Chaixanien, Claire Cardie et al.CVPR 2026 · 4 citations
  3. VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality EvaluationShi-Xue Zhang, Hongfa Wang, Duojun Huang, Xin Li et al.AAAI 2026 · 5 citations
  4. VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual AnalysisShubhashis Roy Dipta, Tz-Ying Wu, Subarna TripathiACL 2026
  5. One Prompt To Rule Them All: LLMs for Opinion Summary EvaluationTejpalsingh Siledar, Swaroop Nath, Sankara Sri Raghava Ravindra Muddu, Rupasai Rangaraju et al.ACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines