Profiling-Guided Bayesian Optimization of JVM Configurations

Regression testing is essential for maintaining software quality but often incurs substantial time costs. While regression testing time can often be reduced by selectively running fewer tests, prior work has demonstrated that tuning Java Virtual Machine (JVM) configuration flags can also reduce testing time in Java projects, even while running all tests and preserving original testing outcomes. However, finding effective flag combinations remains challenging due to the vast configuration space and complex interactions between flags. Random search and direct modeling approaches that map flag configurations to testing time have shown limited effectiveness in navigating this complex optimization landscape. We present PROBO (PROfiling-Guided Bayesian Optimization), an iterative approach that leverages JVM runtime metrics (e.g., garbage collection frequency, just-in-time (JIT) compilation rates, and memory allocation) to guide Bayesian optimization for testing time reduction. Unlike prior work using Bayesian inference that directly models the relationship from flag configurations to testing time, PROBO decomposes the prediction problem through observable runtime behaviors: a metrics model predicts how flag configurations affect runtime metrics, and a performance model predicts how those metrics affect testing time. PROBO collects 44 runtime metrics using a profiler during test execution, and propagates feature importance scores through both models to identify which flags most strongly influence testing time-predictive metrics. Finally, PROBO generates candidate flag configurations through three complementary strategies guided by expected testing time improvement. We evaluate PROBO on 16 open-source Java projects, comparing against random search and BOCA (a Bayesian optimization baseline). PROBO achieves an average testing time reduction of 10.7% across all projects when evaluating 20 configurations per project, outperforming random search (5.8%) by 1.85× and BOCA (4.3%) by 2.49×. PROBO successfully generates configurations that substantially reduce testing time for all 16 projects, with reductions ranging up to 27.4%. With a one-hour time budget for search, PROBO maintains its advantage with 8.0% average reduction, demonstrating practical applicability. PROBO-generated configurations remain effective across software evolution, maintaining 8.8% average reduction over an average of 88 future commits per project. Our analysis reveals that metrics related to JIT compilation, particularly Total Compilation Rate (methods compiled per second) and C1 Compilation Rate (first-tier JIT compilation rate), are the strongest predictors of testing time, accounting for 67.2% of consistently important metrics across projects.