Lune

ACL Findings2024

Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint

Zhipeng Chen, Kun Zhou, Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, Ji-Rong Wen

Abstract

Reinforcement learning (RL) has been widely used in training large language models (LLMs) for preventing unexpected outputs, e.g., reducing harmfulness and errors. However, existing RL methods mainly adopt instance-level reward, which cannot provide fine-grained supervision for complex reasoning tasks. As a result, the RL training cannot be fully aware of the specific part or step that actually leads to the incorrectness in model response. To address it, we propose a new RL method named RLMEC that incorporates a generative model as the reward model, which is trained by the erroneous solution rewriting task under the minimum editing constraint, which can produce token-level supervision for RL training. Based on the generative reward model, we design the token-level RL objective for training and an imitation-based regularization for stabilizing RL process. And these two objectives focus on the revision of the key tokens for the erroneous solution, reducing the effect of other unimportant tokens. Experiment results on 8 tasks have demonstrated the effectiveness of our approach. Our code and data will be publicly released.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Dynamic and Generalizable Process Reward Modeling cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Dynamic and Generalizable Process Reward Modeling cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Dynamic and Generalizable Process Reward Modeling cites Training language models to follow instructions with human feedback
  • T-REG: Preference Optimization with Token-Level Reward Regularization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Large Language Models are Zero-Shot Reasoners cites Training language models to follow instructions with human feedback
  • Text2Grad: Reinforcement Learning from Natural Language Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Text2Grad: Reinforcement Learning from Natural Language Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Text2Grad: Reinforcement Learning from Natural Language Feedback cites Training language models to follow instructions with human feedback
  • Reinforcing LLM Agents via Policy Optimization with Action Decomposition cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Reinforcing LLM Agents via Policy Optimization with Action Decomposition cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • Reinforcing LLM Agents via Policy Optimization with Action Decomposition cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Reinforcing LLM Agents via Policy Optimization with Action Decomposition cites Training language models to follow instructions with human feedback
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
  • SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. SiriuS: Self-improving Multi-agent Systems via Bootstrapped ReasoningWanjia Zhao, Mert Yüksekgönül, Shirley Wu, James Y. ZouNeurIPS 2025 · 49 citations
  2. Reinforcing LLM Agents via Policy Optimization with Action DecompositionMuning Wen, Ziyu Wan, Jun Wang, Weinan Zhang et al.NeurIPS 2024 · 31 citations
  3. Text2Grad: Reinforcement Learning from Natural Language FeedbackHanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao et al.ICLR 2026 · 18 citations
  4. Dynamic and Generalizable Process Reward ModelingZhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Qinyuan Cheng et al.ACL 2025 · 13 citations
  5. T-REG: Preference Optimization with Token-Level Reward RegularizationWenxuan Zhou, Shujian Zhang, Lingxiao Zhao, Tao MengACL 2025 · 11 citations

Builds on31

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  5. Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations

Related papers

  1. Teaching Models to Improve on TapeLiat Bezalel, Eyal Orgad, Amir GlobersonAAAI 2025
  2. On the Generalization of SFT: A Reinforcement Learning Perspective with Reward RectificationYongliang Wu, Yizhou Zhou, Ziheng Zhou, Yingzhe Peng et al.ICLR 2026 · 130 citations
  3. LeTS: Learning to Think-and-Search via Process-and-Outcome Reward HybridizationQi Zhang, Shouqing Yang, Lirong Gao, Hao Chen et al.EMNLP 2025
  4. CARL: Constraint-Aware Reinforcement Learning for Planning with LLMsQiuyi Qi, Jinjian Zhang, Mutian Bao, Tian Liang et al.ACL Findings 2026
  5. Advancing Reasoning in Diffusion Language Models with Denoising Process RewardsShaoan Xie, Lingjing Kong, Xiangchen Song, Xinshuai Dong et al.ACL 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines