ACL Findings2023
UMSE: Unified Multi-scenario Summarization Evaluation
Shen Gao, Zhitao Yao, Chongyang Tao, Xiuying Chen, Pengjie Ren, Zhaochun Ren, Zhumin Chen
Abstract
Summarization quality evaluation is a nontrivial task in text summarization. Contemporary methods can be mainly categorized into two scenarios: (1) reference-based: evaluating with human-labeled reference summary; (2) reference-free: evaluating the summary consistency of the document. Recent studies mainly focus on one of these scenarios and explore training neural models built on pre-trained language models (PLMs) to align with human criteria. However, the models from different scenarios are optimized individually, which may result in sub-optimal performance since they neglect the shared knowledge across different scenarios. Besides, designing individual models for each scenario caused inconvenience to the user. Inspired by this, we propose Unified Multi-scenario Summarization Evaluation Model (UMSE). More specifically, we propose a perturbed prefix tuning method to share cross-scenario knowledge between scenarios and use a self-supervised training paradigm to optimize the model without extra human labeling. Our UMSE is the first unified summarization evaluation framework engaged with the ability to be used in three evaluation scenarios. Experimental results across three typical scenarios on the benchmark dataset SummEval indicate that our UMSE can achieve comparable performance with several existing strong methods which are specifically designed for each scenario. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- BARTScore: Evaluating Generated Text as Text Generation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- BARTScore: Evaluating Generated Text as Text Generation cites BERTScore: Evaluating Text Generation with BERT
- Towards a Unified Multi-Dimensional Evaluator for Text Generation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Towards a Unified Multi-Dimensional Evaluator for Text Generation cites BARTScore: Evaluating Generated Text as Text Generation
- Towards a Unified Multi-Dimensional Evaluator for Text Generation cites BERTScore: Evaluating Text Generation with BERT
- Towards a Unified Multi-Dimensional Evaluator for Text Generation cites FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization cites BERTScore: Evaluating Text Generation with BERT
Builds on12
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- BARTScore: Evaluating Generated Text as Text GenerationWeizhe Yuan, Graham Neubig, Pengfei LiuNeurIPS 2021 · 1,143 citations
- Towards a Unified Multi-Dimensional Evaluator for Text GenerationMing Zhong, Yang Liu, Da Yin, Yuning Mao et al.EMNLP 2022 · 103 citations
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive SummarizationEsin Durmus, He He, Mona T. DiabACL 2020 · 90 citations
Related papers
- UniSumEval: Towards Unified, Fine-grained, Multi-dimensional Summarization Evaluation for LLMsYuho Lee, Taewon Yun, Jason Cai, Hang Su et al.EMNLP Findings 2024 · 4 citations
- UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot SummarizationYulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang et al.ACL 2023 · 7 citations
- An Empirical Study of Cross-Dataset Evaluation for Neural Summarization SystemsYiran Chen, Pengfei Liu, Ming Zhong, Zi-Yi Dou et al.EMNLP Findings 2020 · 28 citations
- Towards Multi-dimensional Evaluation of LLM Summarization across Domains and LanguagesHyangsuk Min, Yuho Lee, Minjeong Ban, Jiaqi Deng et al.ACL 2025 · 8 citations
- UniTE: Unified Translation EvaluationYu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang et al.ACL 2022
