Lune

ACL Findings2025

A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation

Yang Zhong, Diane J. Litman

Abstract

Ensuring factual consistency in summarization remains a challenge, especially for longdocument evaluation.While automated, reference-free evaluation models are essential given the impracticality of large-scale human assessment for lengthy texts, challenges persist in evaluating different systems on how to handle different summary granularities and evolving model generations.In this work, we conduct a systematic study on diverse factualconsistency evaluation systems across four long-document datasets, encompassing summaries generated by models from non-LLMs to proprietary LLMs.Our analysis reveals that fine-grained continuous scores can provide more reliable assessments of different evaluation systems' capabilities than binary classification.We also examine the relationship between sentence-level and summary-level model performance, highlighting its dependency on dataset characteristics.Moreover, our study reveals that advanced systems can achieve higher recall in error detection for older summaries, yet struggle with false positives and fine-grained error detection.Our analysis and case studies provide further insights into designing robust factuality evaluation systems, which are becoming increasingly in demand as generative models advance rapidly.Ex 1: They were all men who paid for their ship with their lives.Label: SentE; Generator: Phi-2; All systems failed to identify the error.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on22

  1. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  3. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
  4. G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
  5. CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive SummarizationShuyang Cao, Lu WangEMNLP 2021 · 130 citations

Related papers

  1. ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code SummarizationSuyoung Bae, CheolWon Na, Jaehoon Lee, Yumin Lee et al.ACL 2026 · 1 citation
  2. Stress Testing Factual Consistency Metrics for Long-Document SummarizationZain Muhammad Mujahid, Dustin Wright, Isabelle AugensteinACL 2026 · 1 citation
  3. How Far are We from Robust Long Abstractive Summarization?Huan Yee Koh, Jiaxin Ju, He Zhang, Ming Liu et al.EMNLP 2022 · 16 citations
  4. Verify with Caution: The Pitfalls of Relying on Imperfect Factuality MetricsAmeya Godbole, Robin JiaACL Findings 2025
  5. Do Automatic Factuality Metrics Measure Factuality? A Critical EvaluationSanjana Ramprasad, Byron C. WallaceNeurIPS 2025 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines