Lune

EMNLP Findings2025

Smart-Searcher: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning

Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Xin Zhao, Lei Fang, Ji-Rong Wen

Abstract

Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the model's internal knowledge. In this paper, we introduce Smart-Searcher, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. Smart-Searcher employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model's internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning. Our experiments demonstrate that Smart-Searcher outperforms previous RAG and reasoning methods and achieves efficient retrieval. The code is available at https:// github.com/RUCAIBox/R1-Searcher-plus.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • RAG-on-a-Diet: A Reinforcement Learning-Based Dynamic Resource Optimization Framework for RAG cites Measuring and Narrowing the Compositionality Gap in Language Models
  • RAG-on-a-Diet: A Reinforcement Learning-Based Dynamic Resource Optimization Framework for RAG cites Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
  • Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites ZeRO: memory optimizations toward training trillion parameter models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. RAG-on-a-Diet: A Reinforcement Learning-Based Dynamic Resource Optimization Framework for RAGHongwen Ding, Yizheng ZhaoACL 2026

Builds on20

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
  3. Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
  4. ZeRO: memory optimizations toward training trillion parameter modelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong HeSC 2020 · 852 citations
  5. Measuring and Narrowing the Compositionality Gap in Language ModelsOfir Press, Muru Zhang, Sewon Min, Ludwig Schmidt et al.EMNLP Findings 2023 · 296 citations

Related papers

  1. CP-Search: A Chain Progressive Search Training Framework Incentivizing the Cognitive Behaviors for Searching in LLMsZehua Wang, Shipeng Li, Buzhou TangAAAI 2026
  2. R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement LearningYuan Li, Qi Luo, Xiaonan Li, Bufan Li et al.EMNLP Findings 2025 · 4 citations
  3. SmartSearch: Process Reward-Guided Query Refinement for Search AgentsTongyu Wen, Guanting Dong, Zhicheng DouSIGIR 2026 · 13 citations
  4. Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented GenerationGuanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang et al.WWW 2025 · 44 citations
  5. Optimizing Retrieval for RAG via Reinforcement LearningJiawei Zhou, Lei ChenNeurIPS 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines