Lune

EMNLP Findings2023

T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics

Yiwei Qin, Weizhe Yuan, Graham Neubig, Pengfei Liu

Abstract

Modern embedding-based metrics for evaluation of generated text generally fall into one of two paradigms: discriminative metrics that are trained to directly predict which outputs are of higher quality according to supervised human annotations, and generative metrics that are trained to evaluate text based on the probabilities of a generative model. Both have their advantages; discriminative metrics are able to directly optimize for the problem of distinguishing between good and bad outputs, while generative metrics can be trained using abundant raw text. In this paper, we present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5SCORE, a metric that uses these training signals with mT5 as backbone. 1 We perform an extensive empirical comparison with other existing metrics on 5 datasets, 19 languages and 280 systems, demonstrating the utility of our method. 2 Experimental results show that: T5SCORE achieves the best performance on all datasets against existing top-scoring metrics at the segment level.

2 Appendix A.9.1 shows the dataset details. 3 We release our code and models at https://anonymous. 4open.science/r/T5Score-F21D.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

8 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Pretraining Language Models Using Translationese cites BARTScore: Evaluating Generated Text as Text Generation
  • Leveraging Large Language Models for NLG Evaluation: Advances and Challenges cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Leveraging Large Language Models for NLG Evaluation: Advances and Challenges cites BARTScore: Evaluating Generated Text as Text Generation
  • Leveraging Large Language Models for NLG Evaluation: Advances and Challenges cites BERTScore: Evaluating Text Generation with BERT
  • BARTScore: Evaluating Generated Text as Text Generation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • BARTScore: Evaluating Generated Text as Text Generation cites BERTScore: Evaluating Text Generation with BERT
  • CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code cites BERTScore: Evaluating Text Generation with BERT
  • BRIO: Bringing Order to Abstractive Summarization cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • BRIO: Bringing Order to Abstractive Summarization cites BARTScore: Evaluating Generated Text as Text Generation
  • BRIO: Bringing Order to Abstractive Summarization cites BERTScore: Evaluating Text Generation with BERT
  • Towards a Unified Multi-Dimensional Evaluator for Text Generation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Towards a Unified Multi-Dimensional Evaluator for Text Generation cites BARTScore: Evaluating Generated Text as Text Generation
  • Towards a Unified Multi-Dimensional Evaluator for Text Generation cites BERTScore: Evaluating Text Generation with BERT
  • MT-Ranker: Reference-free machine translation evaluation by inter-system ranking cites BERTScore: Evaluating Text Generation with BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers8

  1. Document Understanding Dataset and Evaluation (DUDE)Jordy Van Landeghem, Rafal Powalski, Rubèn Tito, Dawid Jurkiewicz et al.ICCV 2023 · 130 citations
  2. CodeBERTScore: Evaluating Code Generation with Pretrained Models of CodeShuyan Zhou, Uri Alon, Sumit Agarwal, Graham NeubigEMNLP 2023 · 77 citations
  3. Leveraging Large Language Models for NLG Evaluation: Advances and ChallengesZhen Li, Xiaohan Xu, Tao Shen, Can Xu et al.EMNLP 2024 · 17 citations
  4. MT-Ranker: Reference-free machine translation evaluation by inter-system rankingIbraheem Muhammad Moosa, Rui Zhang, Wenpeng YinICLR 2024 · 13 citations
  5. Pretraining Language Models Using TranslationeseMeet Doshi, Raj Dabre, Pushpak BhattacharyyaEMNLP 2024

Builds on10

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  3. BARTScore: Evaluating Generated Text as Text GenerationWeizhe Yuan, Graham Neubig, Pengfei LiuNeurIPS 2021 · 1,143 citations
  4. BRIO: Bringing Order to Abstractive SummarizationYixin Liu, Pengfei Liu, Dragomir R. Radev, Graham NeubigACL 2022 · 329 citations
  5. Towards a Unified Multi-Dimensional Evaluator for Text GenerationMing Zhong, Yang Liu, Da Yin, Yuning Mao et al.EMNLP 2022 · 103 citations

Related papers

  1. SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic MistakesWenda Xu, Xian Qian, Mingxuan Wang, Lei Li et al.ACL 2023 · 3 citations
  2. Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error SynthesisWenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon et al.EMNLP Findings 2022 · 9 citations
  3. LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation ScoresYiqi Liu, Nafise Sadat Moosavi, Chenghua LinACL Findings 2024 · 13 citations
  4. Automatic Text Evaluation through the Lens of Wasserstein BarycentersPierre Colombo, Guillaume Staerman, Chloé Clavel, Pablo PiantanidaEMNLP 2021 · 21 citations
  5. DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question AnsweringPei Ke, Fei Huang, Fei Mi, Yasheng Wang et al.ACL 2023 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines