ACL Findings2025
ReasonerRank: Redefining Language Model Evaluation with Ground-Truth-Free Ranking Frameworks
Jiamu Zhang, Jiayi Yuan, Andrew Wen, Hoang Anh Duy Le, Yu-Neng Chuang, Soo-Hyun Choi, Rui Chen, Xia Hu
Abstract
Large Language Models (LLMs) are increasingly adopted across real-world applications, yet traditional evaluations rely on expensive, domain-specific ground-truth labels that are often unavailable or infeasible. We introduce a ground-truth-free evaluation framework focused on reasoning consistency and instruction following, shifting the emphasis from correctness-which is elusive without labels-to transparent, coherent, evidence-based reasoning. Each model response includes a direct answer, a structured multi-step explanation, and supporting evidence, all assessed via semantic similarity and output adherence checks. We further propose TopK-ReRank, which refines rankings by constructing a consensus answer from the most reliable models, reducing ambiguity across diverse reasoning styles. Experiments show that our framework outperforms existing label-free methods, including majority voting, triplet ranking, and peer-review approaches, providing a more interpretable and efficient alternative for evaluating LLMs in the absence of ground-truth labels. Our code is available at https://github.com/MorrisZJ/ ReasonerRank.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Leveraging Large Language Models for NLG Evaluation: Advances and Challenges cites ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- On scalable oversight with weak LLMs judging strong LLMs cites Measuring Massive Multitask Language Understanding
- On scalable oversight with weak LLMs judging strong LLMs cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Measuring Massive Multitask Language Understanding
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
Builds on13
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent DebateChi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu et al.ICLR 2024 · 871 citations
- On scalable oversight with weak LLMs judging strong LLMsZachary Kenton, Noah Y. Siegel, János Kramár, Jonah Brown-Cohen et al.NeurIPS 2024 · 116 citations
- Leveraging Large Language Models for NLG Evaluation: Advances and ChallengesZhen Li, Xiaohan Xu, Tao Shen, Can Xu et al.EMNLP 2024 · 17 citations
Related papers
- Reasoning-Based Refinement of Unsupervised Text Clusters with LLMsTunazzina IslamACL Findings 2026
- Evaluating Step-by-step Reasoning Traces: A SurveyJinu Lee, Julia HockenmaierEMNLP Findings 2025 · 16 citations
- PiCO: Peer Review in LLMs based on Consistency OptimizationKun-Peng Ning, Shuo Yang, Yuyang Liu, Jia-Yu Yao et al.ICLR 2025
- Assessing the Belief Consistency of Large Language Models on the Logical Conversation ProcessTomoki Tsujimura, Matiss Rikters, Masaki Asada, Shusaku Egami et al.ACL 2026
- Reasoning Runtime Behavior of a Program with LLM: How Far are We?Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li et al.ICSE 2025 · 5 citations
