ACL Findings2024
Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM
Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuanjing Huang, Zhongyu Wei
Abstract
How can we construct an automated debate judge to evaluate an extensive, vibrant, multiturn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments. At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate. In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration. To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system's performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation. Source code and benchmark data are available at https: //github.com/ljcleo/debatrix.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment cites Towards a Unified Multi-Dimensional Evaluator for Text Generation
- Strategic Planning and Rationalizing on Trees Make LLMs Better Debaters cites ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation cites G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
- Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation cites Can Large Language Models Be an Alternative to Human Evaluations?
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate cites Towards a Unified Multi-Dimensional Evaluator for Text Generation
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate cites Can Large Language Models Be an Alternative to Human Evaluations?
Cited by top-tier papers6
- Dancing in Chains: Strategic Persuasion in Academic Rebuttal via Theory of MindZhitao He, Zongwei Lyu, Yi R. FungICLR 2026 · 5 citations
- Strategic Planning and Rationalizing on Trees Make LLMs Better DebatersDanqing Wang, Zhuorui Ye, Xinran Zhao, Fei Fang et al.ICLR 2026 · 2 citations
- Debatable Intelligence: Benchmarking LLM Judges via Debate Speech EvaluationNoy Sternlicht, Ariel Gera, Roy Bar-Haim, Tom Hope et al.EMNLP 2025 · 1 citation
- DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate ScenariosTongzhou Yu, Mingjia Li, Hong Qian, Wenkai Wang et al.KDD 2026 · 1 citation
- Conditional Dichotomy Quantification via Geometric EmbeddingShaobo Cui, Wenqing Liu, Yiyang Feng, Jiawei Zhou et al.ACL 2025
Builds on10
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent DebateChi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu et al.ICLR 2024 · 871 citations
- G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
- Can Large Language Models Be an Alternative to Human Evaluations?David Cheng-Han Chiang, Hung-yi LeeACL 2023 · 254 citations
- Towards a Unified Multi-Dimensional Evaluator for Text GenerationMing Zhong, Yang Liu, Da Yin, Yuning Mao et al.EMNLP 2022 · 103 citations
Related papers
- M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation EvaluationZhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren et al.ACL 2025
- SenseJudge: Human-Centric Preference-Driven Judgment FrameworkRui Li, Junfeng Liu, Xiangwen Kong, Zhifang SuiACL Findings 2026
- DEBATE: Devil's Advocate-Based Assessment and Text EvaluationAlex Kim, Keonwoo Kim, Sangwon YoonACL Findings 2024 · 5 citations
- MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn DialoguesGe Bai, Jie Liu, Xingyuan Bu, Yancheng He et al.ACL 2024 · 35 citations
- Conch: Competitive Debate Analysis via Visualizing Clash Points and Hierarchical StrategiesQianhe Chen, Yong Wang, Yixin Yu, Xiyuan Zhu et al.IEEE VIS 2025
