EMNLP Findings2023
DeltaScore: Fine-Grained Story Evaluation with Perturbations
Zhuohan Xie, Miao Li, Trevor Cohn, Jey Han Lau
Abstract
Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness. In this paper, we introduce DELTASCORE, a novel methodology that uses perturbation techniques for the evaluation of nuanced story aspects. We posit that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between preand post-perturbation states using pre-trained language models. We compare DELTASCORE with existing metrics on storytelling datasets from two domains in five fine-grained story aspects: fluency, coherence, relatedness, logicality, and interestingness. DELTASCORE demonstrates strong performance, revealing a surprising finding that one specific perturbation proves highly effective in capturing multiple aspects. Source code is available on our GitHub repository. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- BARTScore: Evaluating Generated Text as Text Generation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- BARTScore: Evaluating Generated Text as Text Generation cites BERTScore: Evaluating Text Generation with BERT
- What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation cites BARTScore: Evaluating Generated Text as Text Generation
- What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation cites BERTScore: Evaluating Text Generation with BERT
- What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation cites ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- Are LLM-based Evaluators Confusing NLG Quality Criteria? cites BERTScore: Evaluating Text Generation with BERT
- FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning cites BERTScore: Evaluating Text Generation with BERT
- EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generation cites ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate cites Language Models are Few-Shot Learners
Cited by top-tier papers4
- FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial ReasoningZhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan et al.ACL 2026 · 13 citations
- Are LLM-based Evaluators Confusing NLG Quality Criteria?Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang et al.ACL 2024 · 6 citations
- What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story EvaluationDingyi Yang, Qin JinACL 2025 · 2 citations
- EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance GenerationXinda Wang, Zhengxu Hou, Yangshijie Zhang, Bingren Yan et al.ACL 2026 · 1 citation
Builds on24
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- BARTScore: Evaluating Generated Text as Text GenerationWeizhe Yuan, Graham Neubig, Pengfei LiuNeurIPS 2021 · 1,143 citations
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent DebateChi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu et al.ICLR 2024 · 871 citations
Related papers
- StoryER: Automatic Story Evaluation via Ranking, Rating and ReasoningHong Chen, Duc Minh Vo, Hiroya Takamura, Yusuke Miyao et al.EMNLP 2022 · 3 citations
- Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error SynthesisWenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon et al.EMNLP Findings 2022 · 9 citations
- OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation MetricsJian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu et al.ACL 2021
- DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question AnsweringPei Ke, Fei Huang, Fei Mi, Yasheng Wang et al.ACL 2023 · 2 citations
- On the Blind Spots of Model-Based Evaluation Metrics for Text GenerationTianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar et al.ACL 2023 · 10 citations
