RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLMas-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training. Original A direct approach (Ram et al., 2023; Shi et al., 2024) to building RALMs involves leveraging the in-context learning of LLMs to generate responses based on the retrieved documents. However, this plug-and-play method may cause the model to generate unfaithful responses or become distracted by noise. Recent works (Asai et al., 2024a; Lin et al., 2024; Yu et al., 2024c) have proposed constructing specialized RAG datasets and applying supervised fine-tuning (SFT) to further increase the usability of RALMs. However, these SFT-based methods may cause RALMs to overly rely on and fit training data, lacking a feedback mechanism that enables the model to capture human preferences. As shown in Figure 1(a), the SFT RALM may cite satirical content from the internet and generate harmful responses, or provide responses that lack sufficient information and fail to fully address the user's needs. To better integrate human preferences like helpful and harmless (Bai et al., 2022) into RALMs, we argue that RALMs should shift towards a new train-