On the Hyperparameter Loss Landscapes of Machine Learning Models: An Exploratory Study

Previous efforts on hyperparameter optimization (HPO) of machine learning (ML) models predominately focus on algorithmic advances, yet little is known about the topography of the underlying hyperparameter (HP) loss landscape, which plays a fundamental role in governing the search process of HPO. While several works have conducted fitness landscape analysis (FLA) on various ML systems, they are limited to properties of isolated landscape without interrogating the potential structural similarities among landscapes induced on different scenarios. The exploration of such similarities can provide a novel perspective for understanding the mechanism behind modern HPO methods, but has been missing. In this paper, we mapped 1,500 HP loss landscapes of 6 representative ML models on 63 datasets across different fidelity levels, with 11M+ configurations. By conducting exploratory analysis on these landscapes with fine-grained visualizations and dedicated FLA metrics, we observed a similar landscape topography across a wide range of models, datasets, and fidelities, and shed light on the mechanism behind the success of several popular methods in HPO. The artifacts associated with this paper is available at https://github.com/COLA-Laboratory/GraphFLA.