Lune

ACL Findings2024

FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

Alessandro Scirè, Karim Ghonim, Roberto Navigli

Abstract

Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance.However, a notable challenge persists as a substantial number of automaticallygenerated summaries exhibit factual inconsistencies, such as hallucinations.In response to this issue, various approaches for the evaluation of consistency for summarization have emerged.Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLMbased metrics.To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric.FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary.Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation.Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization.In the hope of fostering research in summarization factuality evaluation, we release the code of our metric and our factuality annotations of long-form summarization at https://github.com/Babelscape/FENICE. SUMMARY England number eight BillyVunipola is set to return from injury for the Six Nations opener against Wales at Twickenham.0.6 -0.7

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document cites Adversarial NLI: A New Benchmark for Natural Language Understanding
  • FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document cites BERTScore: Evaluating Text Generation with BERT
  • Learning to Reason for Hallucination Span Detection cites FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document
  • Stress Testing Factual Consistency Metrics for Long-Document Summarization cites FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document
  • Stress Testing Factual Consistency Metrics for Long-Document Summarization cites Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
  • Asking and Answering Questions to Evaluate the Factual Consistency of Summaries cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Asking and Answering Questions to Evaluate the Factual Consistency of Summaries cites BERTScore: Evaluating Text Generation with BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Learning to Reason for Hallucination Span DetectionHsuan Su, Ting-Yao Hu, Hema Swetha Koppula, Kundan Krishna et al.ICLR 2026 · 8 citations
  2. Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via EntailmentNguyen Linh Bao Nguyen, Wanlun Ma, Viet Vo, Alsharif Abuadbba et al.USENIX Security 2026 · 4 citations
  3. FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out DocumentJoonho Yang, Seunghyun Yoon, Byeongjeong Kim, Hwanhee LeeEMNLP 2024 · 3 citations
  4. ARGUS: Hallucination and Omission Evaluation in Video-LLMsRuchit Rawal, Reza Shirkavand, Heng Huang, Gowthami Somepalli et al.ICCV 2025 · 1 citation
  5. Stress Testing Factual Consistency Metrics for Long-Document SummarizationZain Muhammad Mujahid, Dustin Wright, Isabelle AugensteinACL 2026 · 1 citation

Builds on19

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
  5. Asking and Answering Questions to Evaluate the Factual Consistency of SummariesAlex Wang, Kyunghyun Cho, Mike LewisACL 2020 · 317 citations

Related papers

  1. Summary Factual Inconsistency Detection Based on LLMs Enhanced by Universal Information ExtractionAnguo Li, Lei YuACL Findings 2025 · 1 citation
  2. Evaluating the Factual Consistency of Large Language Models Through News SummarizationDerek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan et al.ACL Findings 2023 · 78 citations
  3. ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code SummarizationSuyoung Bae, CheolWon Na, Jaehoon Lee, Yumin Lee et al.ACL 2026 · 1 citation
  4. A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization EvaluationYang Zhong, Diane J. LitmanACL Findings 2025 · 1 citation
  5. FaStFact: Faster, Stronger Long-Form Factuality Evaluations in LLMsYingjia Wan, Haochen Tan, Xiao Zhu, Xinyu Zhou et al.EMNLP Findings 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines