Lune

EMNLP Findings2024

Compare without Despair: Reliable Preference Evaluation with Generation Separability

Sayan Ghosh, Tejas Srinivasan, Swabha Swayamdipta

Abstract

Human evaluation of generated language through pairwise preference judgments is pervasive. However, under common scenarios, such as when generations from a model pair are very similar, or when stochastic decoding results in large variations in generations, it results in inconsistent preference ratings. We address these challenges by introducing a metaevaluation measure, SEPARABILITY, which estimates how suitable a test instance is for pairwise preference evaluation. For a candidate test instance, SEPARABILITY samples multiple generations from a pair of models, and measures how distinguishable the two sets of generations are. Our experiments show that instances with high SEPARABILITY values yield more consistent preference ratings from both humanand auto-raters. Further, the distribution of SEPARABILITY allows insights into which test benchmarks are more valuable for comparing models. Finally, we incorporate SEPARABIL-ITY into ELO ratings, accounting for how suitable each test instance might be for reliably ranking LLMs. Overall, SEPARABILITY has implications for consistent, efficient and robust preference evaluation of LLMs with both human-and auto-raters.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • How Reliable is Language Model Micro-Benchmarking? cites Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through EditsTuhin Chakrabarty, Philippe Laban, Chien-Sheng WuCHI 2025 · 14 citations
  2. How Reliable is Language Model Micro-Benchmarking?Gregory Yauney, Shahzaib Saqib Warraich, Swabha SwayamdiptaICLR 2026 · 7 citations
  3. What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token PatternsMichael A. Hedderich, Anyi Wang, Raoyuan Zhao, Florian Eichin et al.ACL 2025

Builds on12

  1. Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos et al.ICML 2024 · 1,212 citations
  2. The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, Albert Webson et al.ICML 2023 · 908 citations
  3. LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi FengNeurIPS 2024 · 865 citations
  4. G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
  5. Abductive Commonsense ReasoningChandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi et al.ICLR 2020 · 521 citations

Related papers

  1. Correct Looks Better: Pairwise Comparisons Reveal Accuracy RankingsMina Remeli, Moritz HardtICML 2026
  2. Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI CombatRoland Daynauth, Christopher Clarke, Krisztián Flautner, Lingjia Tang et al.ACL 2025
  3. Rating Roulette: Self-Inconsistency in LLM-As-A-Judge FrameworksRajarshi Haldar, Julia HockenmaierEMNLP Findings 2025 · 11 citations
  4. Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy CompetitionKehua Feng, Keyan Ding, Hongzhi Tan, Kede Ma et al.ACL 2025
  5. A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better InterpretabilityXinyu Hu, Mingqi Gao, Li Lin, Zhenghan Yu et al.ACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines