Lune

EMNLP Findings2024

AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation

Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

Abstract

Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyperparameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel twolevel Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 ≈ 0.8 for scenarios with prominent gradients in search space, using only ∼ 20% of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation cites Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Demystifying and Enhancing the Efficiency of Large Language Model Based Search AgentsTiannuo Yang, Zebin Yao, Bowen Jin, Lixiao Cui et al.ICLR 2026 · 9 citations
  2. METIS: Fast Quality-Aware RAG Systems with Configuration AdaptationSiddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du et al.SOSP 2025 · 3 citations

Builds on12

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
  4. Hybrid LLM: Cost-Efficient and Quality-Aware Query RoutingDujian Ding, Ankur Mallick, Chi Wang, Robert Sim et al.ICLR 2024 · 282 citations
  5. Enabling Large Language Models to Generate Text with CitationsTianyu Gao, Howard Yen, Jiatong Yu, Danqi ChenEMNLP 2023 · 152 citations

Related papers

  1. A Comparison of Independent and Joint Fine-tuning Strategies for Retrieval-Augmented GenerationNeal Lawton, Alfy Samuel, Anoop Kumar, Daben LiuEMNLP Findings 2025
  2. HAMMER: An Automatic RAG Tuning System via Hierarchical Memory-Guided Monte Carlo Tree SearchYingli Zhou, Zixuan Wang, Yixiang FangSIGMOD 2026 · 1 citation
  3. Adapting to Non-Stationary Environments: Multi-Armed Bandit Enhanced Retrieval-Augmented Generation on Knowledge GraphsXiaqiang Tang, Jian Li, Nan Du, Sihong XieAAAI 2025 · 12 citations
  4. Enhancing RAG Efficiency with Adaptive Context CompressionShuyu Guo, Shuo Zhang, Zhaochun RenEMNLP Findings 2025 · 5 citations
  5. ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented GeneratorJunda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin et al.EMNLP 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines