QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, Kai-Wei Chang
Abstract
Language agents have become a promising solution to complex interactive tasks. One of the key ingredients to the success of language agents is the reward model on the trajectory of the agentic workflow, which provides valuable guidance during training or inference. However, due to the lack of annotations of intermediate interactions, most existing works use an outcome reward model to optimize policies across entire trajectories. This may lead to sub-optimal policies and hinder the overall performance. To address this, we propose QLASS (Q-guided Language Agent Stepwise Search), to automatically generate annotations by estimating Q-values in a stepwise manner for open language agents. By introducing a exploration tree and performing process reward modeling, QLASS provides effective intermediate guidance for each step. With the stepwise guidance, we propose a Q-guided generation strategy to enable language agents to better adapt to longterm value, resulting in significant performance improvement during model inference on complex interactive agent tasks. Notably, even with almost half the annotated data, QLASS retains strong performance, demonstrating its efficiency in handling limited supervision. We also empirically show that QLASS can lead to more effective decision making through qualitative analysis.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- AgentRM: Enhancing Agent Generalization with Reward Modeling cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- AgentRM: Enhancing Agent Generalization with Reward Modeling cites Reflexion: language agents with verbal reinforcement learning
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Let's Verify Step by Step cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Weak-to-Strong Generalization with Failure Trajectories cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Weak-to-Strong Generalization with Failure Trajectories cites ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search
- Weak-to-Strong Generalization with Failure Trajectories cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- SAND: Boosting LLM Agents with Self-Taught Action Deliberation cites Reflexion: language agents with verbal reinforcement learning
- SAND: Boosting LLM Agents with Self-Taught Action Deliberation cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning cites Let's Verify Step by Step
Cited by top-tier papers8
- Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement LearningWenlin Zhang, Xiangyang Li, Kuicai Dong, Yichao Wang et al.NeurIPS 2025 · 85 citations
- AgentRM: Enhancing Agent Generalization with Reward ModelingYu Xia, Jingru Fan, Weize Chen, Siyu Yan et al.ACL 2025 · 20 citations
- Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal AgentsTianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen et al.ACL 2025 · 13 citations
- Weak-to-Strong Generalization with Failure TrajectoriesRuimeng Ye, Zihan Wang, Yang Xiao, Zinan Ling et al.ICLR 2026 · 1 citation
- SAND: Boosting LLM Agents with Self-Taught Action DeliberationYu Xia, Yiran Shen, Junda Wu, Tong Yu et al.EMNLP 2025 · 1 citation
Builds on19
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree SearchDan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue et al.NeurIPS 2024 · 527 citations
Related papers
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value ModelsYuanzhao Zhai, Tingkai Yang, Kele Xu, Dawei Feng et al.AAAI 2025 · 28 citations
- AgentPro: Enhancing LLM Agents with Automated Process SupervisionYuchen Deng, Shichen Fan, Naibo Wang, Xinkui Zhao et al.EMNLP 2025 · 2 citations
- Language Models can Self-Improve at State-Value Estimation for Better SearchEthan Mendes, Alan RitterNeurIPS 2025 · 5 citations
- Boosting Policy and Process Reward Models with Monte Carlo Tree Search in Open-Domain QAChi-Min Chan, Chunpu Xu, Junqi Zhu, Jiaming Ji et al.ACL Findings 2025
- Self-evolving LLM agents with in-distribution OptimizationYudi Zhang, Meng Fang, Zhenfang Chen, Mykola PechenizkiyICML 2026 · 1 citation
