EMNLP Findings2022
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis
Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon, Lei Li, William Yang Wang
Abstract
Is it possible to build a general and automatic natural language generation (NLG) evaluation metric? Existing learned metrics either perform unsatisfactorily or are restricted to tasks where large human rating data is already available. We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation, by utilizing a novel, iterative error synthesis and severity scoring pipeline. This pipeline applies a series of plausible errors to raw text and assigns severity labels by simulating human judgements with entailment. We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings. SESCORE outperforms all prior unsupervised metrics on multiple diverse NLG tasks including machine translation, image captioning, and WebNLG text generation. For WMT 20/21 En-De and Zh-En, SESCORE improve the average Kendall correlation with human judgement from 0.154 to 0.195. SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no humanannotated training data. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback cites BLEURT: Learning Robust Metrics for Text Generation
- INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback cites Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
- INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback cites Toward Human-Like Evaluation for Natural Language Generation with Error Analysis
- INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback cites COMET: A Neural Framework for MT Evaluation
- Polos: Multimodal Metric Learning from Human Feedback for Image Captioning cites INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback
- Polos: Multimodal Metric Learning from Human Feedback for Image Captioning cites BLEURT: Learning Robust Metrics for Text Generation
- Polos: Multimodal Metric Learning from Human Feedback for Image Captioning cites COMET: A Neural Framework for MT Evaluation
- Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing cites BLEURT: Learning Robust Metrics for Text Generation
- Perturbation CheckLists for Evaluating NLG Evaluation Metrics cites BLEURT: Learning Robust Metrics for Text Generation
- On the Blind Spots of Model-Based Evaluation Metrics for Text Generation cites BLEURT: Learning Robust Metrics for Text Generation
- On the Blind Spots of Model-Based Evaluation Metrics for Text Generation cites Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
- On the Blind Spots of Model-Based Evaluation Metrics for Text Generation cites COMET: A Neural Framework for MT Evaluation
- Toward Human-Like Evaluation for Natural Language Generation with Error Analysis cites BLEURT: Learning Robust Metrics for Text Generation
- Toward Human-Like Evaluation for Natural Language Generation with Error Analysis cites Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
- Toward Human-Like Evaluation for Natural Language Generation with Error Analysis cites COMET: A Neural Framework for MT Evaluation
- COMET: A Neural Framework for MT Evaluation cites BLEURT: Learning Robust Metrics for Text Generation
Cited by top-tier papers6
- INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic FeedbackWenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song et al.EMNLP 2023 · 36 citations
- Polos: Multimodal Metric Learning from Human Feedback for Image CaptioningYuiga Wada, Kanta Kaneda, Daichi Saito, Komei SugiuraCVPR 2024 · 16 citations
- On the Blind Spots of Model-Based Evaluation Metrics for Text GenerationTianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar et al.ACL 2023 · 10 citations
- Toward Human-Like Evaluation for Natural Language Generation with Error AnalysisQingyu Lu, Liang Ding, Liping Xie, Kanjian Zhang et al.ACL 2023 · 10 citations
- Multilingual Conceptual Coverage in Text-to-Image ModelsMichael Saxon, William Yang WangACL 2023 · 7 citations
Builds on5
- BLEURT: Learning Robust Metrics for Text GenerationThibault Sellam, Dipanjan Das, Ankur P. ParikhACL 2020 · 40 citations
- Perturbation CheckLists for Evaluating NLG Evaluation MetricsAnanya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan et al.EMNLP 2021 · 32 citations
- Automatic Machine Translation Evaluation in Many Languages via Zero-Shot ParaphrasingBrian Thompson, Matt PostEMNLP 2020 · 7 citations
- COMET: A Neural Framework for MT EvaluationRicardo Rei, Craig Stewart, Ana C. Farinha, Alon LavieEMNLP 2020 · 6 citations
- Modeling Disclosive Transparency in NLP Application DescriptionsMichael Saxon, Sharon Levy, Xinyi Wang, Alon Albalak et al.EMNLP 2021 · 3 citations
Related papers
- SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic MistakesWenda Xu, Xian Qian, Mingxuan Wang, Lei Li et al.ACL 2023 · 3 citations
- RoMe: A Robust Metric for Evaluating Natural Language GenerationMd Rashad Al Hasan Rony, Liubov Kovriguina, Debanjan Chaudhuri, Ricardo Usbeck et al.ACL 2022 · 15 citations
- Language Model Augmented Relevance ScoreRuibo Liu, Jason Wei, Soroush VosoughiACL 2021
- Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement TheoryZiang Xiao, Susu Zhang, Vivian Lai, Q. Vera LiaoEMNLP 2023 · 6 citations
- LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis EvaluationYujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang et al.NeurIPS 2023 · 119 citations
