Lune

ACL Findings2025

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao

摘要

Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLMas-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 9 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Self-Rewarding Language Models 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Making Retrieval-Augmented Language Models Robust to Irrelevant Context 引用了 Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences 引用了 Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper9

  1. RewardBench 2: Advancing Reward Model EvaluationSaumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison 等ICLR 2026 · 139 次引用
  2. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgeDawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi 等EMNLP 2025 · 37 次引用
  3. Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal AgentsTianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen 等ACL 2025 · 13 次引用
  4. Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form PreferencesZhuoran Jin, Hongbang Yuan, Kejian Zhu, Jiachun Li 等ICLR 2026 · 11 次引用
  5. Long-form RewardBench: Evaluating Reward Models for Long-form GenerationHui Huang, Yancheng He, Wei Liu, Muyun Yang 等AAAI 2026 · 1 次引用

它引用的 Paper38

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni 等NeurIPS 2020 · 19,162 次引用
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning 等NeurIPS 2023 · 10,924 次引用
  3. Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat 等ICML 2020 · 2,937 次引用
  4. Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li 等ICML 2024 · 569 次引用
  5. Making Retrieval-Augmented Language Models Robust to Irrelevant ContextOri Yoran, Tomer Wolfson, Ori Ram, Jonathan BerantICLR 2024 · 361 次引用

相关 Paper

  1. RAGferee: Building Contextual Reward Models for Retrieval-Augmented GenerationAndrei Catalin Coman, Ionut-Teodor Sorodoc, Leonardo F. R. Ribeiro, Bill Byrne 等EMNLP 2025
  2. M-RewardBench: Evaluating Reward Models in Multilingual SettingsSrishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary 等ACL 2025
  3. RMB: Comprehensively benchmarking reward models in LLM alignmentEnyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi 等ICLR 2025
  4. RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented GenerationShi-Qi Yan, Quan Liu, Zhen-Hua LingACL 2025 · 4 次引用
  5. RubricBench: Aligning Model-Generated Rubrics with Human StandardsJunyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu 等ACL 2026 · 7 次引用
黄昏的海面,两侧是细线勾勒的悬崖