Lune

ICML2025Top-tier venue

R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models

Pengyi Li, Jianye Hao, Hongyao Tang, Yifu Yuan, Jinbin Qiao, Zibin Dong, Yan Zheng

Abstract

Reward functions are crucial for policy learning. Large Language Models (LLMs), with strong coding capabilities and valuable domain knowledge, provide an automated solution for high-quality reward design. However, code-based reward functions require precise guiding logic and parameter configurations within a vast design space, leading to low optimization efficiency. To address the challenges, we propose an efficient automated reward design framework, called R*, which decomposes reward design into two parts: reward structure evolution and parameter alignment optimization. To design high-quality reward structures, R* maintains a reward function population and modularizes the functional components. LLMs are employed as the mutation operator, and modulelevel crossover is proposed to facilitate efficient exploration and exploitation. To design more efficient reward parameters, R* first leverages LLMs to generate multiple critic functions for trajectory comparison and annotation. Based on these critics, a voting mechanism is employed to collect the trajectory segments with high-confidence labels. These labeled segments are then used to refine the reward function parameters through preference learning. Experiments on diverse robotic control tasks demonstrate that R* outperforms strong baselines in both reward design efficiency and quality, surpassing human-designed reward functions. Recently, Large Language Models (LLMs) have demon-

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Eureka: Human-Level Reward Design via Coding Large Language Models cites The Perils of Trial-and-Error Reward Design: Misdesign through Overfitting and Invalid Task Specifications
  • CORE: Collaborative Optimization with Reinforcement Learning and Evolutionary Algorithm for Floorplanning cites Eureka: Human-Level Reward Design via Coding Large Language Models
  • Text2Reward: Reward Shaping with Language Models for Reinforcement Learning cites PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
  • LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search cites Eureka: Human-Level Reward Design via Coding Large Language Models
  • LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search cites PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
  • LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search cites Text2Reward: Reward Shaping with Language Models for Reinforcement Learning
  • On Discovering Algorithms for Adversarial Imitation Learning cites Eureka: Human-Level Reward Design via Coding Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward SearchPengyi Li, Hongyao Tang, Jinbin Qiao, Yan Zheng et al.NeurIPS 2025 · 7 citations
  2. CORE: Collaborative Optimization with Reinforcement Learning and Evolutionary Algorithm for FloorplanningPengyi Li, Shixiong Kai, Jianye Hao, Ruizhe Zhong et al.NeurIPS 2025 · 1 citation
  3. On Discovering Algorithms for Adversarial Imitation LearningShashank Reddy Chirra, Jayden Teoh, Praveen Paruchuri, Pradeep VarakanthamICLR 2026 · 1 citation

Builds on13

  1. Eureka: Human-Level Reward Design via Coding Large Language ModelsYecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang et al.ICLR 2024 · 582 citations
  2. PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingKimin Lee, Laura M. Smith, Pieter AbbeelICML 2021 · 380 citations
  3. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative SimulationYufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang et al.ICML 2024 · 227 citations
  4. Text2Reward: Reward Shaping with Language Models for Reinforcement LearningTianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu et al.ICLR 2024 · 142 citations
  5. The Perils of Trial-and-Error Reward Design: Misdesign through Overfitting and Invalid Task SpecificationsSerena Booth, W. Bradley Knox, Julie Shah, Scott Niekum et al.AAAI 2023 · 103 citations

Related papers

  1. Learning Reward for Robot Skills Using Large Language Models via Self-AlignmentYuwei Zeng, Yao Mu, Lin ShaoICML 2024 · 26 citations
  2. RF-Agent: Automated Reward Function Design via Language Agent Tree SearchNing Gao, Xiuhui Zhang, Xingyu Jiang, Mukang You et al.NeurIPS 2025 · 8 citations
  3. Efficient Language-instructed Skill Acquisition via Reward-Policy Co-EvolutionChangxin Huang, Yanbin Chang, Junfan Lin, Junyang Liang et al.AAAI 2025 · 1 citation
  4. REvolve: Reward Evolution with Large Language Models using Human FeedbackRishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi et al.ICLR 2025
  5. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model FeedbackYufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian et al.ICML 2024 · 135 citations
Dusk over the sea between two cliffs drawn in fine vertical lines