Chain-of-Retrieval Augmented Generation
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
Abstract
This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Conventional RAG methods usually perform a single retrieval step before the generation process, which limits their effectiveness in addressing complex queries due to imperfect retrieval results. In contrast, our proposed method, CoRAG (Chain-of-Retrieval Augmented Generation), allows the model to dynamically reformulate the query based on the evolving state. To train CoRAG effectively, we utilize rejection sampling to automatically generate intermediate retrieval chains, thereby augmenting existing RAG datasets that only provide the correct final answer. At test time, we propose various decoding strategies to scale the model's test-time compute by controlling the length and number of sampled retrieval chains. Experimental results across multiple benchmarks validate the efficacy of CoRAG, particularly in multi-hop question answering tasks, where we observe more than 10 points improvement in EM score compared to strong baselines. On the KILT benchmark, CoRAG establishes a new state-of-the-art performance across a diverse range of knowledge-intensive tasks. Furthermore, we offer comprehensive analyses to understand the scaling behavior of CoRAG, laying the groundwork for future research aimed at developing factual and grounded foundation models.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
16 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- STaR: Bootstrapping Reasoning With Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- WebThinker: Empowering Large Reasoning Models with Deep Research Capability cites Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- WebThinker: Empowering Large Reasoning Models with Deep Research Capability cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- WebThinker: Empowering Large Reasoning Models with Deep Research Capability cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- DeepRAG: Thinking to Retrieve Step by Step for Large Language Models cites Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- DeepRAG: Thinking to Retrieve Step by Step for Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Cited by top-tier papers16
- WebThinker: Empowering Large Reasoning Models with Deep Research CapabilityXiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian et al.NeurIPS 2025 · 354 citations
- StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy OptimizationXuhui Zheng, Kang An, Ziliang Wang, Yuhang Wang et al.EMNLP 2025 · 41 citations
- DeepRAG: Thinking to Retrieve Step by Step for Large Language ModelsXinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin et al.ICLR 2026 · 30 citations
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning ChainsXuying Ning, Dongqi Fu, Tianxin Wei, Mengting Ai et al.ICLR 2026 · 14 citations
- HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented GenerationPeilin Wu, Mian Zhang, Kun Wan, Wentian Zhao et al.ICLR 2026 · 13 citations
Builds on18
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
- STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
Related papers
- TRACE the Evidence: Constructing Knowledge-Grounded Reasoning Chains for Retrieval-Augmented GenerationJinyuan Fang, Zaiqiao Meng, Craig MacDonaldEMNLP Findings 2024 · 23 citations
- KiRAG: Knowledge-Driven Iterative Retriever for Enhancing Retrieval-Augmented GenerationJinyuan Fang, Zaiqiao Meng, Craig MacDonaldACL 2025
- DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question AnsweringRong Cheng, Jinyi Liu, Yan Zheng, Fei Ni et al.ACL 2025
- FrugalRAG: Less is More in RL Finetuning for Multi-hop Question AnsweringAbhinav Java, Srivathsan Koundinyan, Nagarajan Natarajan, Amit SharmaICLR 2026 · 2 citations
- HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented GenerationYihan Jiao, Zhehao Tan, Dan Yang, Duolin Sun et al.EMNLP Findings 2025 · 1 citation
