EMNLP Findings2025
Smart-Searcher: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning
Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Xin Zhao, Lei Fang, Ji-Rong Wen
Abstract
Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the model's internal knowledge. In this paper, we introduce Smart-Searcher, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. Smart-Searcher employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model's internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning. Our experiments demonstrate that Smart-Searcher outperforms previous RAG and reasoning methods and achieves efficient retrieval. The code is available at https:// github.com/RUCAIBox/R1-Searcher-plus.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- RAG-on-a-Diet: A Reinforcement Learning-Based Dynamic Resource Optimization Framework for RAG cites Measuring and Narrowing the Compositionality Gap in Language Models
- RAG-on-a-Diet: A Reinforcement Learning-Based Dynamic Resource Optimization Framework for RAG cites Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites ZeRO: memory optimizations toward training trillion parameter models
Cited by top-tier papers1
Builds on20
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
- Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
- ZeRO: memory optimizations toward training trillion parameter modelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong HeSC 2020 · 852 citations
- Measuring and Narrowing the Compositionality Gap in Language ModelsOfir Press, Muru Zhang, Sewon Min, Ludwig Schmidt et al.EMNLP Findings 2023 · 296 citations
Related papers
- CP-Search: A Chain Progressive Search Training Framework Incentivizing the Cognitive Behaviors for Searching in LLMsZehua Wang, Shipeng Li, Buzhou TangAAAI 2026
- R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement LearningYuan Li, Qi Luo, Xiaonan Li, Bufan Li et al.EMNLP Findings 2025 · 4 citations
- SmartSearch: Process Reward-Guided Query Refinement for Search AgentsTongyu Wen, Guanting Dong, Zhicheng DouSIGIR 2026 · 13 citations
- Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented GenerationGuanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang et al.WWW 2025 · 44 citations
- Optimizing Retrieval for RAG via Reinforcement LearningJiawei Zhou, Lei ChenNeurIPS 2025 · 1 citation
