Multi-Agent Debate for LLM Judges with Adaptive Stability Detection

With the advancing reasoning capabilities of Large Language Models (LLMs), they are increasingly employed for complex evaluation tasks, such as grading student responses, verifying factual claims, and comparing competing answers. Leveraging multiple LLMs as automated judges can enhance robustness and accuracy by aggregating diverse perspectives, yet existing approaches often rely on static and simple aggregation methods, such as majority voting, which may produce incorrect judgments despite correct individual assessments. We propose a novel multiagent debate framework where LLMs collaboratively reason and iteratively refine judgments, formalizing this process mathematically and proving its advantages over static ensembles. To ensure computational efficiency, we introduce a stability detection mechanism using a time-varying Beta-Binomial mixture model (a mixture of two Beta-Binomial distributions) that tracks judge consensus dynamics and applies adaptive stopping via Kolmogorov-Smirnov testing. Experiments across diverse benchmarks and models demonstrate significant improvements in judgment accuracy over majority voting while maintaining computational efficiency.

LLMs-as-Judges. Our work is closely related to the field of LLM-as-a-Judge [Zheng et al., 2023, Gu et al., 2025] and LLMs-as-Judges [Li et al., 2024], which involves using one or more LLMs to evaluate responses generated by either another LLM or a human. Basic LLM-as-Judge frameworks typically rely on a single LLM to perform a judgement task [