Lost in the Context: Insufficient and Distracted Attention to Contexts in Preference Modeling

In Reinforcement Learning from Human Feedback (RLHF), the reward model (RM) evaluates the response quality based on the given context and assigns a reward. It plays a crucial role in aligning RLHF with human preferences. Although the current RM training paradigm concatenates the context and response while amplifying the reward difference between good and bad response pairs, we demonstrate that the RM faces two significant issues: i) it often allocates only a small proportion of attention to the context, and ii) it frequently ignores segments of the context that are relevant for evaluating the response quality. These issues undermine the RM's effectiveness in modeling human preferences. To further address these challenges, we propose AttnRM, a novel optimization framework that enables the RM to concentrate on crucial segments of the context. Experimental results demonstrate that At-tnRM significantly improves preference modeling by increasing attention to relevant information within the context. It also enhances the RM's generalizability and achieves better performance in aligning with human preferences. * Equal contribution. † Corresponding author. 0.35 0.39 Reward = 2.87 (High) 0.17 Total attention score: 0.09 RM Relevant segments Irrelevant segments Normalized Attention Score Distribution Containing key information for evaluating response quality Noise segments that is irrelevant to response Segment attention score: > 0.06 Segment attention score: < 0.03 … <BOS> <EOS> <EOS> Response Context 0.31 0.42 Reward = 2.95 (Remain high) 0.20