Profile-guided System Optimizations for Accelerated Greybox Fuzzing

Greybox fuzzing is a highly popular option for security testing, incentivizing tremendous efforts to improve its performance. Prior research has brought many algorithmic advancements, leading to substantial performance growth. However, less attention has been paid to the system-level designs of greybox fuzzing tools, despite the high impacts of such designs on fuzzing throughput. In this paper, we explore system-level optimizations for greybox fuzzing. Throughout an empirical study, we unveil two systemlevel optimization opportunities. First, the common fuzzing mode with a fork server visibly slows down the target execution, which can be optimized by coupling persistent mode with efficient state recovery. Second, greybox fuzzing tools rely on the native Operating System (OS) to support interactions issued by the target program, involving complex but fuzzing-irrelevant operations. Simplification of OS interactions represents another optimization opportunity. We develop two techniques, informed by a short profiling phase of the fuzzing tool, to achieve the optimizations above. The first technique enables reliable and efficient persistent mode by learning critical execution states from the profiling and patching the target program to reset them. The second technique introduces user-space abstractions to simulate OS functionality, reducing expensive OS interactions. Evaluated with 20 programs and the MAGMA benchmark, we demonstrate that our optimizations can accelerate AFL and AFL++ for higher code coverage and faster bug finding. CCS CONCEPTS • Security and privacy → Software and application security.