Incremental Program Analysis in the Wild: An Empirical Study on Real-World Program Changes

Incremental program analysis (IPA) has gained increasing attention as an effective approach for maintaining up-to-date analysis results by leveraging previously computed results in response to program changes. Consequently, a variety of IPA algorithms and tools have been proposed. However, their empirical performance in practical, real-world scenarios remains insufficiently investigated. To address this gap, this study presents a comprehensive examination of the current state-of-the-art in IPA evaluation. Specifically, we identify two key limitations: (1) the lack of standardized benchmarks reflecting real-world program changes, and (2) the inadequacy and imbalanced distribution of evaluation metrics.To overcome these challenges, we propose an automated pipeline for constructing real-world program change benchmarks and develop a unified incremental evaluation framework for systematically evaluating IPA tools. Using the proposed evaluation pipeline, we constructed large-scale benchmarks of real-world program changes—sourced from 4,084 commits across 20 Java projects—and systematically evaluated two IPA tools for Java. The results demonstrate that, although incremental analysis substantially improves efficiency compared to exhaustive analysis, existing IPA tools exhibit inconsistencies and markedly higher peak memory consumption. Finally, we distill practical insights from our findings to inform future research and development in the field of incremental program analysis.