Generative Reward Modeling via Synthetic Criteria Preference Learning
Xiaobo Liang, Haoke Zhang, Juntao Li, Kehai Chen, Qiaoming Zhu, Min Zhang
Abstract
Generative Reward Models (GenRMs) leverage synthesized Chains of Thought (CoT) to reduce the need for massive labeled data, but this approach introduces risks of overoptimization due to the inability to guarantee the correctness of the CoTs. Identifying and optimizing unexpected behaviors within these synthesized CoT remains a challenge, as it heavily depends on precise annotations of intermediate behavior, similar to process supervision. In this work, we introduce a criteria-based preference tree for reward modeling, where each path in the tree represents a reasoning trajectory based on synthesized criteria. Crucially, each reasoning trajectory can be independently optimized through RL algorithm. These fine-grained process reward signals are derived from the inference-time computations and predefined rules, eliminating the need for human supervision. In experiments, SyncPL 1 showed significant improvements over baselines on multiple human preference benchmarks. We further demonstrate that synthesized data can be learned us-ing a long CoT format, analogous to an o1-like model, further enhancing performance while keeping stability and efficiency during training.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- GRAM-R²: Self-Training Generative Foundation Reward Models for Reward Reasoning cites AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
- GRAM-R²: Self-Training Generative Foundation Reward Models for Reward Reasoning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Cited by top-tier papers3
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM AlignmentTianci Liu, Ran Xu, Tony Yu, Ilgee Hong et al.ACL 2026 · 75 citations
- GRAM-R²: Self-Training Generative Foundation Reward Models for Reward ReasoningChenglong Wang, Yongyu Mu, Hang Zhou, Yifu Huo et al.AAAI 2026 · 5 citations
- DUAL RM: Beyond Rule-based Preference Reward Modeling via Meta-RewardXiaobo Liang, Wanfu Wang, Qipeng Huang, Yuyang Ding et al.ACL 2026
Builds on18
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human FeedbackYann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang et al.NeurIPS 2023 · 948 citations
- ZeRO: memory optimizations toward training trillion parameter modelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong HeSC 2020 · 852 citations
Related papers
- GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative ReasoningJian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou et al.AAAI 2026 · 68 citations
- Think-RM: Enabling Long-Horizon Reasoning in Generative Reward ModelsIlgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan et al.NeurIPS 2025 · 15 citations
- Smarter Not Harder: Generative Process Evaluation with Intrinsic-Signal Driving and Ability‑Adaptive Reward ShapingTao He, Rongchuan Mu, Lizi Liao, Yixin Cao et al.ICLR 2026
- Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-TuningYibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang et al.NeurIPS 2025 · 102 citations
- Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference OptimizationBin Hong, Jiayu Liu, Kai Zhang, Jianwen Sun et al.ICLR 2026 · 1 citation
