Lune

ACL Findings2025

Removing Prompt-template Bias in Reinforcement Learning from Human Feedback

Chaojie Wang, Haonan Shi, Long Tian, Bo An, Shuicheng Yan

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become an essential technique for enhancing pre-trained large language models (LLMs) to generate responses that align with human preferences and societal values. Although RLHF has shown promise, the training of reward models (RMs) still faces the challenge of reward hacking, motivating recent works to prevent RMs from finding shortcuts that bypass the intended optimization objectives by identifying simplistic patterns such as response length. Besides the issue of length bias, our work firstly reveals that prompt-template bias learned by RMs can also cause reward hacking when dealing with some marginal samples, resulting in LLMs preferring to generate responses in a specific format after RLHF fine-tuning, regardless of the format requested in the prompt. To this end, we propose a low-cost but effective method, namely Prompt Bias Calibration (PBC), to estimate the prompt-template bias term during reward modeling, which can be utilized to calibrate reward scores in the following RL fine-tuning process. Then, we show that our PBC method can be flexibly combined with existing algorithms of removing length bias, leading to a further improvement in the aspect of enhancing the quality of generated responses.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Model Alignment as Prospect Theoretic Optimization cites Measuring Massive Multitask Language Understanding
  • Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on8

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  4. Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
  5. Scaling Laws for Reward Model OveroptimizationLeo Gao, John Schulman, Jacob HiltonICML 2023 · 963 citations

Related papers

  1. ODIN: Disentangled Reward Mitigates Hacking in RLHFLichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia et al.ICML 2024 · 119 citations
  2. Bias Fitting to Mitigate Length Bias of Reward Model in RLHFKangwen Zhao, Jianfeng Cai, Jinhua Zhu, Ruopei Sun et al.ACL 2026 · 7 citations
  3. Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human FeedbackWei Shen, Guanlin Liu, Yu Yue, Ruofei Zhu et al.NeurIPS 2025 · 33 citations
  4. Mitigating Length Bias in RLHF Through a Causal LensHyeonji Kim, Sujeong Oh, Sanghack LeeAAAI 2026 · 3 citations
  5. Post-hoc Reward Calibration: A Case Study on Length BiasZeyu Huang, Zihan Qiu, Zili Wang, Edoardo M. Ponti et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines