GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search

Enhancing large models for complex multihop question-answering has become a research focus in the Retrieval-augmented generation (RAG) area. Many existing approaches aim to mimic human thought processes by enabling large models to perform retrieval-augmented generation step by step. However, these methods can only perform single chain reasoning, which lacks the ability for multi-path exploration, strategic look-ahead, stepwise evaluation, and global selection. In addition, to effectively decompose complex problems, these methods can only rely on labor-intensive intermediate annotations for supervised fine-tuning. To address these issues, we propose GRAT, an algorithm guided by Monte Carlo Tree Search (MCTS) and process rewards. GRAT not only enables self-evaluation and self-correction but also assigns fine-grained rewards to each intermediate step in the search path. These finegrained annotations can be used for model selftraining, which enables GRAT to continuously self-update its problem analysis and reasoning capabilities. We conducted experiments on four multihop QA datasets: HotPotQA, 2WikiMul-tiHopQA, MuSiQue, and Bamboogle, demonstrating that GRAT outperforms various RAGbased methods. Additionally, incorporating self-training significantly enhances GRAT's reasoning performance. 1