Rethinking the Role of Proxy Rewards in Language Model Alignment
Abstract
Learning from human feedback via proxy reward modeling has been studied to align Large Language Models (LLMs) with human values. However, achieving reliable training through that proxy reward model (RM) is not a trivial problem, and its behavior remained as a blackbox. In this paper, we study the role of proxy rewards in the LLM alignment via 'reverse reward engineering' by composing interpretable features as a white-box reward function. We aim to replicate the ground truth (gold) reward signal by achieving a monotonic relationship between the proxy and gold reward signals after training the model using the proxy reward in reinforcement learning (RL). Our findings indicate that successfully emulating the gold reward requires generating responses that are relevant with enough length to open-ended questions, while also ensuring response consistency in closed-ended questions. Furthermore, resulting models optimizing our devised white-box reward show competitive performances with strong open-source RMs in alignment benchmarks. We highlight its potential usage as a simple but strong reward baseline for the LLM alignment, not requiring explicit pairwise human feedback dataset and RM training 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- MPO: Multilingual Safety Alignment via Reward Gap Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Towards Reward Fairness in RLHF: From a Resource Allocation Perspective cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Towards Reward Fairness in RLHF: From a Resource Allocation Perspective cites Training language models to follow instructions with human feedback
Cited by top-tier papers2
Builds on24
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
Related papers
- Aligning Large Language Models through Synthetic FeedbackSungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang et al.EMNLP 2023 · 13 citations
- Interpreting Language Reward Models via Contrastive ExplanationsJunqi Jiang, Tom Bewley, Saumitra Mishra, Freddy Lécué et al.ICLR 2025
- Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model AlignmentYuang Cai, Yuyu Yuan, Jinsheng Shi, Qinhong LinAAAI 2025 · 5 citations
- Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-ExpertsHaoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao et al.EMNLP Findings 2024 · 17 citations
- Preference Learning for AI Alignment: a Causal PerspectiveKasia Kobalczyk, Mihaela van der SchaarICML 2025
