ACL Findings2026
Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation
Adam Dejl, James Barry, Alessandra Pascale, Javier Carnerero-Cano
Abstract
Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce outputs that are incomplete or selectively omit key information. In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies, including hallucinations. In this study, we address the challenge of evaluating the comprehensiveness of LLM-generated texts, focusing on the detection of missing information or underrepresented viewpoints. We investigate three automated evaluation metrics:
(1) an NLI-based method that decomposes texts into atomic statements and uses natural language inference (NLI) to identify missing facts, (2) a Q&A-based metric that extracts questionanswer pairs and compares responses across sources, and (3) an end-to-end approach that directly identifies missing content using LLMs.
Our experiments demonstrate the surprising effectiveness of the simple end-to-end metric compared to more complex metrics, though at the cost of reduced robustness, interpretability and result granularity. We further assess the comprehensiveness of responses from several popular open-weight LLMs when answering user queries based on multiple sources.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on10
- Asking and Answering Questions to Evaluate the Factual Consistency of SummariesAlex Wang, Kyunghyun Cho, Mike LewisACL 2020 · 317 citations
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive SummarizationEsin Durmus, He He, Mona T. DiabACL 2020 · 90 citations
- The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language ModelsJunyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng et al.ACL 2024 · 49 citations
- Factuality of Large Language Models: A SurveyYuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu et al.EMNLP 2024 · 23 citations
- VeriScore: Evaluating the factuality of verifiable claims in long-form text generationYixiao Song, Yekyung Kim, Mohit IyyerEMNLP Findings 2024 · 15 citations
Related papers
- ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive SummarizationDavid Wan, Koustuv Sinha, Srini Iyer, Asli Celikyilmaz et al.ACL Findings 2024 · 11 citations
- FineSurE: Fine-grained Summarization Evaluation using LLMsHwanjun Song, Hang Su, Igor Shalyminov, Jason Cai et al.ACL 2024
- Evaluating LLMs' Assessment of Mixed-Context Hallucination Through the Lens of SummarizationSiya Qi, Rui Cao, Yulan He, Zheng YuanACL Findings 2025
- Exploring Precision and Recall to assess the quality and diversity of LLMsFlorian Le Bronnec, Alexandre Verine, Benjamin Négrevergne, Yann Chevaleyre et al.ACL 2024 · 11 citations
- How Much Do LLMs Hallucinate across Languages? On Realistic Multilingual Estimation of LLM HallucinationSaad Obaid ul Islam, Anne Lauscher, Goran GlavasEMNLP 2025
