CombCT: Compiler Testing via Combinatorial Testing

Compilers, as fundamental software, are essential for various applications, making their reliability crucial. Modern compilers like GCC and LLVM offer numerous optimizations to improve performance. Empirical studies present that most bugs occur during compiler optimization, especially with specific optimization sequences, yet little attention has been paid to this in compiler testing. The vast optimization sequence space challenges existing methods, leaving this important research direction in its early stages and in need of effective solutions. In this work, we propose a novel and effective compiler testing approach dubbed CombCT, aimed at boosting compiler testing. CombCT is the first approach that leverages the power of combinatorial testing for compiler testing, making it be capable of effectively exploring optimization sequences. Besides, CombCT incorporates three new techniques, i.e., dynamic optimization sequence adaption, partial sequence selection, and diversity augmentation, to enhance its effectiveness. Extensive experiments on various versions of GCC and LLVM demonstrate that CombCT detects much more unique bugs, which are caused by certain optimization sequences, than current state-of-the-art methods, indicating its superiority. Moreover, CombCT has disclosed 14 previously unknown bugs in the latest versions of GCC and LLVM, 13 of which have been confirmed or fixed, showing its practical value.