J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization

To keep pace with the increasing pace of large language models (LLM) development, model output evaluation has transitioned away from time-consuming human evaluation to automatic evaluation, where LLMs themselves are tasked with assessing and critiquing other model outputs. LLM-as-judge models are a class of generative evaluators that excel in evaluating relatively simple domains, like chat quality, but struggle in reasoning intensive domains where model responses contain more substantive and challenging content. To remedy existing judge shortcomings, we explore training judges with reinforcement learning (RL). We make three key contributions: (1) We propose the Equivalent Initial State Group Relative Policy Optimization (EIS-GRPO) algorithm, which allows us to train our judge to be robust to positional biases that arise in more complex evaluation settings. (2) We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work. (3) We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%, matching or exceeding the performance of larger GRPO-trained judges on both JudgeBench and ReasoningJudgeBench. : GitHub : ReasoningJudgeBench Reward + Advantage Training w/ transformation invariant RL algorithm Benchmark: Enhanced evaluation of judges in reasoning settings Methodology: EIS-GRPO for robust judge training Model: J4R-7B, a small yet effective reasoning evaluator Figure 1: Overview of our three contributions: (1) We propose EIS-GRPO, a RL training algorithm for training positionally robust judge models. (2) We create ReasoningJudgeBench, a benchmark of 1,483 pairwise samples across diverse and underexplored reasoning settings. (3) We train J4R-7B, a lightweight yet high-performing judge specifically for reasoning evaluation. et al. [2024]). Furthermore, automatic evaluation outputs have a limited set of final outcomes (e.g., "A" or "B" for pairwise comparisons, 1-5 for single ratings), making them naturally "verifiable". Before simply applying standard RL algorithms [Shao et al., 2024 , Guo et al., 2025] with judge data, we first take a step back and carefully look into where judges fail in reasoning settings. Our analysis in Sec. 4 shows that as task difficulty grows, existing judge models, including those trained with RL, suffer from increasing positional inconsistency, where swapping the order of the candidates does not lead to an equivalent swap in judgment. This inconsistency indicates that the judge is randomly guessing, rather than substantively assessing. To remedy this, we propose Equivalent Initial State Group Relative Policy Optimization (EIS-GRPO), which explicitly trains the model to treat transformations of the same initial state (i.e., input context) as equivalent, instilling consistency and robustness in judges with zero additional training overhead. Using EIS-GRPO, we train Judge for Reasoning (J4R), a 7B LLM judge that specializes in evaluating reasoning. J4R outperforms GPT-4o by 6.7% and approaches the performance of 32B RL-trained judges on a suite of reasoning-focused judge benchmarks. In our evaluation, we find existing benchmarks of reasoning judges lack breadth. Notably, JudgeBench [Tan et al., 2024] consists only of 350 pairwise samples sourced from MMLU-Pro [Wang et al., 2024b], LiveBench [White et al., 2024], and LiveCodeBench [Jain et al., 2024], missing coverage of different, uniquely challenging reasoning tasks, like inductive or commonsense reasoning. This leads us to create ReasoningJudgeBench to augment existing benchmarks with new, challenging types of reasoning. As shown in Fig. 1, our contributions are threefold: • Methodology (Sec. 3): We propose Equivalent Initial State GRPO, a simple but effective data augmentation method for GRPO that allows us to combat judge positional bias while improving performance over data-driven approaches. • Benchmark (Sec. 4.1): We create ReasoningJudgeBench, a benchmark of 1,483 challenging pairwise samples. Compared to existing reasoning-focused judge benchmarks, ReasoningJudgeBench contains 4x more samples covering more diverse reasoning settings. • Model and Analysis (Sec. 5): Using EIS-GRPO, we train J4R-CJ-7B, a 7B model that punches above its weight class: It matches the performance of larger judge models in reasoning evaluation and outperforms the next best ≤ 14B judge by 13% and 9% on JudgeBench and Reasoning-JudgeBench. We also perform a suite of analyses focused on (1) the utility of RL versus other common judge training recipes, (2) choice of initial model, and (3) the efficacy of EIS-GRPO versus other bias-mitigation training strategies.