Lune

ICLR2026Top-tier venue

J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason E. Weston, Ilia Kulikov, Swarnadeep Saha

Abstract

The progress of AI is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models a core solution. The efficacy of these judges depends on their chain-of-thought reasoning, creating a critical need for methods that can effectively optimize this reasoning process. In this work, we introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance across multiple benchmarks. In particular, J1-Qwen-32B, our multitasked pointwise and pairwise judge also outperforms o1-mini, o3, and a much larger 671B DeepSeek-R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effectiveness of our approach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dynamic criteria generation, reference answer creation, iterative self-correction of initial assessments, and feedback generation for low-quality responses.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

23 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge cites Training language models to follow instructions with human feedback
  • WildChat: 1M ChatGPT Interaction Logs in the Wild cites Training language models to follow instructions with human feedback
  • RM-R1: Reward Modeling as Reasoning cites Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
  • Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains cites RM-R1: Reward Modeling as Reasoning
  • RM-R1: Reward Modeling as Reasoning cites Training language models to follow instructions with human feedback
  • Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains cites Training language models to follow instructions with human feedback
  • Online Rubrics Elicitation from Pairwise Comparisons cites Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
  • Online Rubrics Elicitation from Pairwise Comparisons cites Training language models to follow instructions with human feedback
  • Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy cites RM-R1: Reward Modeling as Reasoning
  • Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers23

  1. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable DomainsAnisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath et al.ICLR 2026 · 340 citations
  2. Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI SynergyChris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He et al.ICLR 2026 · 211 citations
  3. Online Rubrics Elicitation from Pairwise ComparisonsMohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang et al.ICML 2026 · 41 citations
  4. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgeDawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi et al.EMNLP 2025 · 37 citations
  5. Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and ImageYushi Hu, Reyhane Askari Hemmat, Melissa Hall, Emily Dinan et al.CVPR 2026 · 18 citations

Builds on19

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  3. WildChat: 1M ChatGPT Interaction Logs in the WildWenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie et al.ICLR 2024 · 504 citations
  4. Prometheus: Inducing Fine-Grained Evaluation Capability in Language ModelsSeungone Kim, Jamin Shin, Yejin Choi, Joel Jang et al.ICLR 2024 · 468 citations
  5. RM-R1: Reward Modeling as ReasoningXiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin et al.ICLR 2026 · 147 citations

Related papers

  1. Think-J: Learning to Think for Generative LLM-as-a-JudgeHui Huang, Yancheng He, Hongli Zhou, Rui Zhang et al.AAAI 2026 · 12 citations
  2. J4R: Learning to Judge with Equivalent Initial State Group Relative Policy OptimizationAustin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong et al.ACL 2026 · 8 citations
  3. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding TasksHongchao Jiang, Yiming Chen, Yushi Cao, Hung-Yi Lee et al.ACL 2026 · 33 citations
  4. ReSearch: Learning to Reason with Search for LLMs via Reinforcement LearningMingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun et al.NeurIPS 2025 · 125 citations
  5. Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement LearningRan Xu, Jingjing Chen, Jiayu Ye, Yu Wu et al.ICLR 2026 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines