MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models
Yujing Wang, Hainan Zhang, Liang Pang, Binghui Guo, Hongwei Zheng, Zhiming Zheng
Abstract
In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts, necessitating query rewriting to better describe user's information needs. However, traditional context-based rewriting has minimal enhancement on downstream generation tasks due to the lengthy process from query rewriting to response generation. Some researchers try to utilize reinforcement learning with generation feedback to assist the rewriter, but this sparse rewards provide little guidance in most cases, leading to unstable training and generation results.We find that user's needs are also reflected in the gold documents, retrieved documents and ground-truth. Therefore, by feeding back these multi-aspect dense rewards to query rewriting, more stable and satisfactory responses can be achieved. In this paper, we propose a novel query rewriting method MaFeRw, which improves RAG performance by integrating multi-aspect feedback from both the retrieval process and generated results. Specifically, we first use manual data to train a T5 model for the rewriter initialization. Next, we design three metrics as reinforcement learning feedback: the similarity between the rewritten query and the gold document, the ranking metrics, and ROUGE between the generation and the ground truth. Inspired by RLAIF, we train three kinds of reward models for the above metrics to achieve more efficient training. Finally, we combine the scores of these reward models as feedback, and use PPO algorithm to explore the optimal query rewriting strategy.Experimental results on two conversational RAG datasets demonstrate that MaFeRw achieves superior generation metrics and more stable training compared to baselines.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search cites CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning
- Connect the Dots: Knowledge Graph–Guided Crawler Attack on Retrieval-Augmented Generation Systems cites Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generation
- Connect the Dots: Knowledge Graph–Guided Crawler Attack on Retrieval-Augmented Generation Systems cites Query Rewriting in Retrieval-Augmented Large Language Models
- Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generation cites Query Rewriting in Retrieval-Augmented Large Language Models
- Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting cites CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning
- Query Rewriting in Retrieval-Augmented Large Language Models cites Toolformer: Language Models Can Teach Themselves to Use Tools
Cited by top-tier papers7
- AgenticScholar: Agentic Data Management with Pipeline Orchestration for Scholarly CorporaHai Lan, Tingting Wang, Zhifeng Bao, Guoliang Li et al.SIGMOD 2026 · 4 citations
- Connect the Dots: Knowledge Graph–Guided Crawler Attack on Retrieval-Augmented Generation SystemsMengyu Yao, Ziqi Zhang, Ning Luo, Shaofei Li et al.USENIX Security 2026 · 3 citations
- RESTL: Reinforcement Learning Guided by Multi-Aspect Rewards for Signal Temporal Logic TransformationYue Fang, Zhi Jin, Jie An, Hongshen Chen et al.AAAI 2026 · 1 citation
- Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented GenerationAli Naseh, Yuefeng Peng, Anshuman Suri, Harsh Chaudhari et al.CCS 2025
- WikiREVIEW: A Multi-Perspective Review Framework for Automatic Wiki-Style Article GenerationGuo-Biao Zhang, Zhijing Wu, Tian Lan, Ding-Yuan Liu et al.AAAI 2026
Builds on10
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- Query Rewriting in Retrieval-Augmented Large Language ModelsXinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao et al.EMNLP 2023 · 191 citations
- Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational SearchKelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou et al.EMNLP Findings 2023 · 57 citations
- CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement LearningZeqiu Wu, Yi Luan, Hannah Rashkin, David Reitter et al.EMNLP 2022 · 37 citations
- Enhancing Conversational Search: Large Language Model-Aided Informative Query RewritingFanghua Ye, Meng Fang, Shenghui Li, Emine YilmazEMNLP Findings 2023 · 35 citations
Related papers
- RaFe: Ranking Feedback Improves Query Rewriting for RAGShengyu Mao, Yong Jiang, Boli Chen, Xiao Li et al.EMNLP Findings 2024 · 19 citations
- DVCQR: Dual-View Conversational Query Rewriting with Stage-wise Reinforcement LearningChenyi Li, Xinhui Tu, Zaixiang WangACL 2026
- Generalized Pseudo-Relevance FeedbackYiteng Tu, Weihang Su, Yujia Zhou, Yiqun Liu et al.WWW 2026 · 2 citations
- Think Then Rewrite: Reasoning Enhanced Query Rewriting for Domain Specific RetrievalAng Li, Yufei Shi, Yuxuan Si, Yiquan Wu et al.AAAI 2026
- Dialogue-RAG: Enhancing Retrieval for LLMs via Node-Linking Utterance RewritingQiwei Li, Teng Xiao, Zuchao Li, Ping Wang et al.ACL 2025
