Lune

ACL Findings2025

Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation

Chris Samarinas, Alexander Krubner, Alireza Salemi, Youngwoo Kim, Hamed Zamani

Abstract

This paper presents ICAT, 1 an evaluation framework for measuring coverage of diverse factual information in long-form text generation. ICAT breaks down a long output text into a list of atomic claims and not only verifies each claim through retrieval from a (reliable) knowledge source, but also computes the alignment between the atomic factual claims and various aspects expected to be presented in the output. We study three implementations of the ICAT framework, each with a different assumption on the availability of aspects and alignment method. By adopting data from the diversification task in the TREC Web Track and the ClueWeb corpus, we evaluate the ICAT framework. We demonstrate strong correlation with human judgments and provide comprehensive evaluation across multiple state-ofthe-art LLMs. Our framework further offers interpretable and fine-grained analysis of diversity and coverage. Its modular design allows for easy adaptation to different domains and datasets, making it a valuable tool for evaluating the qualitative aspects of long-form responses produced by LLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LaMP-QA: A Benchmark for Personalized Long-form Question Answering cites G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
  • G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment cites BERTScore: Evaluating Text Generation with BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. LaMP-QA: A Benchmark for Personalized Long-form Question AnsweringAlireza Salemi, Hamed ZamaniEMNLP 2025 · 1 citation
  2. Search for Coverage: Learning Coverage-Aware Retrieval with Augmented Sub-Question AnswerabilityJia-Huei Ju, Eugene Yang, Trevor Adriaanse, Suzan Verberne et al.SIGIR 2026

Builds on13

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  4. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
  5. G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations

Related papers

  1. FaStFact: Faster, Stronger Long-Form Factuality Evaluations in LLMsYingjia Wan, Haochen Tan, Xiao Zhu, Xinyu Zhou et al.EMNLP Findings 2025 · 2 citations
  2. Long-form factuality in large language modelsJerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu et al.NeurIPS 2024 · 182 citations
  3. VeriScore: Evaluating the factuality of verifiable claims in long-form text generationYixiao Song, Yekyung Kim, Mohit IyyerEMNLP Findings 2024 · 15 citations
  4. IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model GenerationHaozhi Fan, Jinhao Duan, Kaidi XuACL 2026 · 1 citation
  5. An Analysis of Multilingual FActScoreVu Trong Kim, Michael Krumdick, Varshini Reddy, Franck Dernoncourt et al.EMNLP 2024 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines