Lune

ACL Findings2025

ReasonerRank: Redefining Language Model Evaluation with Ground-Truth-Free Ranking Frameworks

Jiamu Zhang, Jiayi Yuan, Andrew Wen, Hoang Anh Duy Le, Yu-Neng Chuang, Soo-Hyun Choi, Rui Chen, Xia Hu

Abstract

Large Language Models (LLMs) are increasingly adopted across real-world applications, yet traditional evaluations rely on expensive, domain-specific ground-truth labels that are often unavailable or infeasible. We introduce a ground-truth-free evaluation framework focused on reasoning consistency and instruction following, shifting the emphasis from correctness-which is elusive without labels-to transparent, coherent, evidence-based reasoning. Each model response includes a direct answer, a structured multi-step explanation, and supporting evidence, all assessed via semantic similarity and output adherence checks. We further propose TopK-ReRank, which refines rankings by constructing a consensus answer from the most reliable models, reducing ambiguity across diverse reasoning styles. Experiments show that our framework outperforms existing label-free methods, including majority voting, triplet ranking, and peer-review approaches, providing a more interpretable and efficient alternative for evaluating LLMs in the absence of ground-truth labels. Our code is available at https://github.com/MorrisZJ/ ReasonerRank.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Leveraging Large Language Models for NLG Evaluation: Advances and Challenges cites ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
  • On scalable oversight with weak LLMs judging strong LLMs cites Measuring Massive Multitask Language Understanding
  • On scalable oversight with weak LLMs judging strong LLMs cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
  • Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Measuring Massive Multitask Language Understanding
  • ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on13

  1. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  2. Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
  3. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent DebateChi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu et al.ICLR 2024 · 871 citations
  4. On scalable oversight with weak LLMs judging strong LLMsZachary Kenton, Noah Y. Siegel, János Kramár, Jonah Brown-Cohen et al.NeurIPS 2024 · 116 citations
  5. Leveraging Large Language Models for NLG Evaluation: Advances and ChallengesZhen Li, Xiaohan Xu, Tao Shen, Can Xu et al.EMNLP 2024 · 17 citations

Related papers

  1. Reasoning-Based Refinement of Unsupervised Text Clusters with LLMsTunazzina IslamACL Findings 2026
  2. Evaluating Step-by-step Reasoning Traces: A SurveyJinu Lee, Julia HockenmaierEMNLP Findings 2025 · 16 citations
  3. PiCO: Peer Review in LLMs based on Consistency OptimizationKun-Peng Ning, Shuo Yang, Yuyang Liu, Jia-Yu Yao et al.ICLR 2025
  4. Assessing the Belief Consistency of Large Language Models on the Logical Conversation ProcessTomoki Tsujimura, Matiss Rikters, Masaki Asada, Shusaku Egami et al.ACL 2026
  5. Reasoning Runtime Behavior of a Program with LLM: How Far are We?Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li et al.ICSE 2025 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines