Lune

ICML2025Top-tier venue

Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation

Zhuohao Yu, Weizheng Gu, Yidong Wang, Xingru Jiang, Zhengran Zeng, Jindong Wang, Wei Ye, Shikun Zhang

Abstract

Large Language Models excel at code generation yet struggle with complex programming tasks that demand sophisticated reasoning. To bridge this gap, traditional process supervision relies on learned reward models requiring costly training data and suffering from reward misalignment, while outcome supervision fails for complex tasks needing coordinated intermediate steps. We introduce Outcome Refining Process Supervision, which unifies process and outcome supervision by leveraging executable verification: a treestructured search framework generates strategic alternatives, profiles execution metrics, and scores candidates via self-critique mechanisms that integrate runtime feedback with reasoning. Experiments across 5 models and 3 benchmarks show consistent gains, with 26.9% higher correctness and 42.2% improved code efficiency. The results demonstrate that ORPS enables LLMs to overcome local optima in code generation, suggesting a promising direction for combining verifiable outcomes with structured reasoning to tackle complex challenges. We open-source at: https: //github.com/zhuohaoyu/ORPS

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge cites Language Models are Few-Shot Learners
  • From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge cites Training language models to follow instructions with human feedback
  • Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
  • From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism cites Training language models to follow instructions with human feedback
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgeDawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi et al.EMNLP 2025 · 37 citations
  2. From Curiosity to Caution: Mitigating Reward Hacking for Best-of-NN with PessimismZhuohao Yu, Steven Z. Wu, Adam BlockICLR 2026 · 3 citations

Builds on43

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  4. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  5. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations

Related papers

  1. Process-Supervised Reinforcement Learning for Code GenerationYufan Ye, Ting Zhang, Wenbin Jiang, Hua HuangEMNLP 2025 · 1 citation
  2. CodePRM: Execution Feedback-enhanced Process Reward Model for Code GenerationQingyao Li, Xinyi Dai, Xiangyang Li, Weinan Zhang et al.ACL Findings 2025 · 6 citations
  3. StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language ModelsChenyu Zhou, Tianyi Xu, Jianghao Lin, Dongdong GeICLR 2026 · 29 citations
  4. Graph Reasoning Paradigm: Structured and Symbolic Reasoning with Topology-Aware Reinforcement Learning for Large Language ModelsRunxuan Liu, Xianhao Ou, Xinyan Ma, Jiyuan Wang et al.ACL 2026
  5. Group Verification-based Policy Optimization for Interactive Coding AgentsSilong Dai, Changzhi Sun, Haolun Wu, Huanran Zheng et al.ICLR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines