Lune

NeurIPS2025Top-tier venue

Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers

Zhengliang Shi, Lingyong Yan, Dawei Yin, Suzan Verberne, Maarten de Rijke, Zhaochun Ren

Abstract

Large language models have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a selfincentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning. To enable LLM with this capability, EXSEARCH adopts a generalized expectation-maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated trajectories, progressively improving itself for search. We theoretically analyze this training process and establish convergence guarantees. Extensive experiments on four benchmarks show that EXSEARCH outperforms baselines substantially, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce EXSEARCH-Zoo, an extension that extends our method to broader scenarios, to facilitate future work. Code is available on EXSEARCH.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal ModelsJitai Hao, Hao Liu, Xinyan Xiao, Qiang Huang et al.ICLR 2026 · 18 citations

Builds on34

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
  5. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language ModelsZixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji et al.ICML 2024 · 527 citations

Related papers

  1. ReSearch: Learning to Reason with Search for LLMs via Reinforcement LearningMingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun et al.NeurIPS 2025 · 125 citations
  2. R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement LearningQingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha et al.ACL Findings 2026
  3. Learning to Retrieve from Agent TrajectoriesYuqi Zhou, Sunhao Dai, Changle Qu, Liang Pang et al.SIGIR 2026
  4. Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic SearchChuzhan Hao, Wenfeng Feng, Guochao Jiang, Guofeng Quan et al.ACL Findings 2026
  5. Beneficial Reasoning Behaviors in Agentic Search and Effective Training Methods to Obtain ThemJiahe Jin, Abhijay Sai Paladugu, Chenyan XiongACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines