HAMMER: An Automatic RAG Tuning System via Hierarchical Memory-Guided Monte Carlo Tree Search

Retrieval-Augmented Generation (RAG) effectively integrates external knowledge into large language models (LLMs), enhancing accuracy, adaptability, interpretability, and trustworthiness. The modern RAG systems contain multiple modules with many tunable parameters, each of which can greatly influence the overall performance of downstream applications. To identify the optimal configuration for a given RAG pipeline, existing studies have relied on hyperparameter optimization (HPO) methods. However, these methods overlook parameter dependencies, lack experience-based learning, and incur substantial computational costs. To tackle those issues, we propose HAMMER, a hierarchical memory–guided Monte Carlo Tree Search (MCTS) system. Inspired by human learning, HAMMER employs a hierarchical graph memory to organize experimental insights and integrates it with MCTS for more reliable tuning. We further design a theoretically guaranteed query selection technique to reduce cost while preserving effectiveness. Extensive experiments on eight real-world datasets show that HAMMER improves exact match by up to 20.0% and F1-score by 15.2%, while reducing both tuning time and token consumption by up to 9×.