Enhancing the Performance of Bandit-based Hyperparameter Optimization

Bandit-based methods are commonly used for hyperparameter optimization (HPO), which is significant in data analytics. When confronted with numerous configurations and high-dimensional large problems, existing bandit-based methods face challenges of high evaluation cost and poor optimization performance. To address these challenges, we introduce an improved bandit-based approach that exhibits enhanced evaluation ability and is suitable for situations with limited resources. Specifically, our method first effectively utilizes the feature and label information to conduct representative groups for further evaluation. After that, two kinds of folds (i.e., general folds and special folds) are constructed to facilitate better evaluation of the configuration in the cross-validation process. Additionally, we incorporate variance and subset size into the evaluation metric to comprehensively evaluate the configuration. We integrate our proposed method into three commonly used bandit-based methods, and experimental results on multiple datasets show that our method has advantages in stability with accuracy improvement of 1% to 15% on the datasets tested. In addition, since our method can avoid configurations that are low-quality but time-consuming to evaluate, it is always more efficient than the existing bandit-based methods, and can even reduce the execution time by half in some datasets. Sometimes it takes a little more time, but the improvement in accuracy can be significant.

• We investigate the challenges faced by bandit-based methods in terms of performance, stability, and time consumption when dealing with a large number of configurations and complex problems. Meanwhile, we further explore and highlight the significant impact of the subset sampling process and the cross-validation process on optimization.

• To overcome these issues, we present a method that utilizes both feature and label information to perform better subset sampling through group construction. Furthermore, we introduce general and specific folds in the cross-validation process to better evaluate configurations. We further integrate variance and subset size into the evaluation metric, which enhances the optimization performance.

• Our experimental and theoretical findings showcase the superiority of our method in terms of performance, stability, and efficiency, especially in scenarios with a large number of configurations. Our proposed method achieves accuracy improvement ranging from 1% to 10%, while consuming less time and exhibiting lower variance. Furthermore, we have successfully employed our techniques in cross-validation experiments and regression problems, which further vali-