LLM4JMH: Studying the Use of LLMs for Generating Java Performance Microbenchmarks

Performance regressions pose a significant threat to software reliability. In recent years, smaller-scale performance testing, such as performance microbenchmarks, has been adopted in practice to detect such regressions in an early stage of development. Developing these microbenchmarks is costly, error-prone, and demands specialized expertise. Motivated by the recent progress of large language models (LLMs) in code-related tasks, we study whether LLMs can step in as performance experts to automate the generation of microbenchmarks for performance regression detection. While existing approaches typically depend on functional unit tests to generate performance microbenchmarks, we instead design an LLM-based approach, LLM4JMH, to assess the capability of LLMs in generating reliable and effective performance microbenchmarks directly from source code. We further explore how program analysis techniques, such as static analysis, can enhance the reliability and effectiveness of LLM-generated performance microbenchmarks. Experimental results show that LLM-generated performance microbenchmarks achieve comparable bug detection effectiveness to expert-written microbenchmarks, while reducing overall expected detection latency by up to 52.60% across RxJava, Eclipse Collections, and Zipkin. In the study, the generated tests can successfully identify six out of 11 real-world performance bugs in Apache Flink. These results demonstrate that LLM-based performance microbenchmark generation can automate early performance regression detection in continuous integration pipelines of the software development process and reduce reliance on expert-crafted tests, advancing performance-aware software engineering.