ACL Findings2026
R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, Bowen Ma, Zhichun Wang, Shijie Jia, Limin Liu, Xin Wang
Abstract
Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoningsearch interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multistep reasoning with deep search interaction, and learn optimal reasoning-search interaction trajectories via multi-reward signals, improving response quality in complex logic-and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to search or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search significantly outperforms mainstream RAG baselines. The code and data are available at https://github.com/QingFei1/ R-Search.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Search Self-Play: Pushing the Frontier of Agent Capability without Supervision cites Language Models are Few-Shot Learners
- Search Self-Play: Pushing the Frontier of Agent Capability without Supervision cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Efficient Memory Management for Large Language Model Serving with PagedAttention cites Language Models are Few-Shot Learners
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Retrieval Augmented Language Model Pre-Training
- Language Models are Few-Shot Learners cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- SmartSearch: Process Reward-Guided Query Refinement for Search Agents cites Language Models are Few-Shot Learners
- SmartSearch: Process Reward-Guided Query Refinement for Search Agents cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Cited by top-tier papers7
- A Survey of Large Language Model-Based Search AgentsYunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou et al.ACL 2026 · 1,216 citations
- Search Self-Play: Pushing the Frontier of Agent Capability without SupervisionHongliang Lu, Yuhang Wen, Pengyu Cheng, Ruijin Ding et al.ICLR 2026 · 35 citations
- TIPS: Turn-level Information-Potential Reward Shaping for Search-Augmented LLMsYutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu et al.ICLR 2026 · 29 citations
- SmartSearch: Process Reward-Guided Query Refinement for Search AgentsTongyu Wen, Guanting Dong, Zhicheng DouSIGIR 2026 · 13 citations
- Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMsKang An, Ziliang Wang, Xuhui Zheng, FaQiang Qian et al.ICLR 2026 · 6 citations
Builds on22
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
Related papers
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement LearningMingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun et al.NeurIPS 2025 · 125 citations
- LeTS: Learning to Think-and-Search via Process-and-Outcome Reward HybridizationQi Zhang, Shouqing Yang, Lirong Gao, Hao Chen et al.EMNLP 2025
- Iterative Self-Incentivization Empowers Large Language Models as Agentic SearchersZhengliang Shi, Lingyong Yan, Dawei Yin, Suzan Verberne et al.NeurIPS 2025 · 15 citations
- Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement LearningWenlin Zhang, Xiangyang Li, Kuicai Dong, Yichao Wang et al.NeurIPS 2025 · 85 citations
- A Survey of RAG-Reasoning Systems in Large Language ModelsYangning Li, Weizhi Zhang, Yuyao Yang, Wei-Chieh Huang et al.EMNLP Findings 2025 · 9 citations
