One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discovery

Managing large programs and fixing bugs is a challenging task that demands substantial time and manual effort. In practice, when a bug is found, it is reported to the project maintainers, who work with the reporter to fix it. Once the bug is resolved, the issue is marked as closed. However, across the program, there are often similar code segments, which may also contain the bug, but were initially missed during discovery. Finding and fixing each recurring bug instance individually is labor intensive. Even more concerning, bug reports can inadvertently widen the attack surface as they provide attackers with an exploitable pattern that may be unresolved in other parts of the program. In this paper, we explore these Recurring Pattern Bugs (RPBs) that appear repeatedly across various code segments of a program or even in different programs, stemming from a same root cause, but are unresolved, even when a fix is available. Our investigation reveals that RPBs are not only prevalent, but also have the potential to significantly compromise the security of software programs. Various static analyzers exist for finding specific bug patterns but require significant engineering effort and don't generalize well beyond their predefined template. To tackle RPBs and enhance project maintenance efficiency, this paper introduces BUGSTONE, a program analysis system empowered by LLVM and a Large Language Model (LLM). The key observation is that many RPBs have a single instance patched, which can be leveraged to identify a consistent error pattern, such as a specific API misuse. By examining the entire program for other instances of this pattern, it is possible to identify similar sections of code that are vulnerable to the same type of bug. Starting with 135 unique seed RPBs, BUGSTONE was able to identify more than 22K new potential issues in the Linux kernel. Manual analysis of 400 of these findings confirmed that 246 are valid issues. These detected issues include invalid pointer dereferences, resource leaks, type errors, performance issues, and others, threatening the security and stability of the program. We also create a dataset from over 1.9K security bugs reported by 23 recent top-tier conference works targeting bug identification in the Linux kernel. We manually annotate the dataset, identify 80 recurring patterns and 850 corresponding fixes, and evaluate BUGSTONE on this dataset across six different LLMs and six prompt configurations. Even with a cost-efficient model choice, BUGSTONE achieved 92.2% precision and 79.1% pairwise accuracy, demonstrating its capability to accurately identify RPBs in real-world programs.