Agent-as-Judge for Factual Summarization of Long Narratives

Large Language Models (LLMs) have demonstrated near-human performance in summarization tasks based on traditional metrics such as ROUGE and BERTScore. However, these metrics do not adequately capture critical aspects of summarization quality, such as factual accuracy, particularly for long narratives (>100K tokens). Recent advances, such as LLM-as-a-Judge, address the limitations of metrics based on lexical similarity but still exhibit factual inconsistencies, especially in understanding character relationships and states. In this work, we introduce NARRATIVEFACTSCORE (NFS), the first "Agent-as-a-Judge" framework that evaluates and refines factuality in narrative summarization. By leveraging a Character Knowledge Graph (CKG) extracted from input narrative, NARRATIVEFACTSCORE evaluates the factuality and provides actionable guidance for refinement, such as identifying missing or erroneous facts. Our experimental results demonstrate that constructing the CKG enables reasoning with 1/3 of the factuality computation used in the prior approach, and achieve three times higher correlation with human judgments. Furthermore, refinement with actionable guidance improves the quality of the summary. 1 * Corresponding Author 1 https://github.com/YeonseokJeong/NarrativeFactScore # 14. BAG END LIVING ROOM … Bilbo: It's mine, my own. my precious (Frodo rushes into Bag End. He stops and picks up the ring at his feet.) … # 25. BAG END KITCHEN … Gandalf: Sauron needs only this ring to cover all the lands in the second darkness. He is seeking it, seeking it, all his thought is bent on it. … Frodo: Alright! ... input story Gandalf warned Frodo, who carries the Ring, that its master is Sauron. Sauron is searching for the Ring and is pursuing Gandalf.