ACL Findings2026
Reward Yourself: Efficient Self Rewards for Trustworthy Sampling
Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang
Abstract
As high-quality data becomes harder to obtain, reward models are increasingly important. Beyond the costly RLHF stage, they are now used at inference time to guide LLM generation and in data selection for post-training. These methods bring efficiency and performance gains, but current reward models often fail to prevent untrustworthy behaviors such as privacy leaks and stereotypes. Re-training reward models to address these issues is expensive, since it requires large-scale human preference data. We propose SelfRW, a lightweight intrinsic reward that needs no extra fine-tuning or auxiliary models. By pruning current LLMs to approximate an "trust" and an "untrust" token distribution, we compute the log-probability difference as an auxiliary reward. When integrated into rewardguided sampling, SelfRW significantly reduces untrustworthy outputs while preserving task performance. It also improves reward-guided data selection, yielding better post-trained models. Experiments with two reward models and four LLMs on privacy, bias, and stereotype benchmarks show that combining SelfRW consistently improves trustworthiness (over 10% in privacy tasks and 20% in bias tasks) with minimal impact on general utility benchmarks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications cites Training language models to follow instructions with human feedback
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory cites Training language models to follow instructions with human feedback
Builds on13
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank ModificationsBoyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie et al.ICML 2024 · 215 citations
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity TheoryNiloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov et al.ICLR 2024 · 198 citations
Related papers
- More RLHF, More Trust? On The Impact of Preference Alignment On TrustworthinessAaron Jiaxun Li, Satyapriya Krishna, Himabindu LakkarajuICLR 2025
- Semi-Supervised Reward Modeling via Iterative Self-TrainingYifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis et al.EMNLP Findings 2024 · 1 citation
- Learning Goal-Conditioned Representations for Language Reward ModelsVaskar Nath, Dylan Slack, Jeff Da, Yuntao Ma et al.NeurIPS 2024 · 2 citations
- A Systematic Analysis of Base Model Choice for Reward ModelingKian Ahrabian, Pegah Jandaghi, Negar Mokhberian, Sai Praneeth Karimireddy et al.EMNLP 2025
- Understanding Reward Hacking in Text-to-Image Reinforcement LearningYunqi Hong, Kuei-Chun Kao, Hengguang Zhou, Cho-Jui HsiehCVPR Findings 2026
