EMNLP Findings2024
AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation
Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
Abstract
Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyperparameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel twolevel Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 ≈ 0.8 for scenarios with prominent gradients in search space, using only ∼ 20% of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
- Demystifying and Enhancing the Efficiency of Large Language Model Based Search AgentsTiannuo Yang, Zebin Yao, Bowen Jin, Lixiao Cui et al.ICLR 2026 · 9 citations
- METIS: Fast Quality-Aware RAG Systems with Configuration AdaptationSiddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du et al.SOSP 2025 · 3 citations
Builds on12
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
- Hybrid LLM: Cost-Efficient and Quality-Aware Query RoutingDujian Ding, Ankur Mallick, Chi Wang, Robert Sim et al.ICLR 2024 · 282 citations
- Enabling Large Language Models to Generate Text with CitationsTianyu Gao, Howard Yen, Jiatong Yu, Danqi ChenEMNLP 2023 · 152 citations
Related papers
- A Comparison of Independent and Joint Fine-tuning Strategies for Retrieval-Augmented GenerationNeal Lawton, Alfy Samuel, Anoop Kumar, Daben LiuEMNLP Findings 2025
- HAMMER: An Automatic RAG Tuning System via Hierarchical Memory-Guided Monte Carlo Tree SearchYingli Zhou, Zixuan Wang, Yixiang FangSIGMOD 2026 · 1 citation
- Adapting to Non-Stationary Environments: Multi-Armed Bandit Enhanced Retrieval-Augmented Generation on Knowledge GraphsXiaqiang Tang, Jian Li, Nan Du, Sihong XieAAAI 2025 · 12 citations
- Enhancing RAG Efficiency with Adaptive Context CompressionShuyu Guo, Shuo Zhang, Zhaochun RenEMNLP Findings 2025 · 5 citations
- ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented GeneratorJunda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin et al.EMNLP 2024 · 6 citations
