ZendDiff: Differential Testing of PHP Interpreter

The PHP interpreter, powering over 70% of websites on the internet, plays a crucial role in web development. Existing approaches to finding bugs in PHP primarily focus on detecting explicit security issues through crashes or sanitizerbased oracles, but fail to identify logic bugs that can silently lead to incorrect results. We observe that the introduction of Just-In-Time (JIT) compilation mode in PHP presents an opportunity for differential testing, as it provides an alternative implementation of the same language specification. We propose, ZendDiff, an automatic differential testing framework that effectively detects logic bugs in the PHP interpreter by comparing JIT and non-JIT execution results. Our differential testing incorporates three techniques: program state probing for fine-grained execution state comparison, JIT-aware program mutation to sufficiently exercise JIT functionality, and dual verification to handle nondeterministic behaviors in PHP programs. Our experimental results demonstrate that ZendDiff outperforms the official test suite used in PHP's continuous integration, achieving higher code coverage and executing more Zend opcodes. Through ablation studies, we validate the effectiveness of these techniques. To date, ZendDiff has identified 51 previously unknown logic bugs in the PHP interpreter, with 37 already fixed and 3 confirmed by the PHP maintainers. ZendDiff has been acknowledged by the PHP community and offers a practical tool for automatically discovering logic bugs in the PHP interpreter. Index Terms-differential testing, just-in-time compilation, logic bug detection, PHP interpreter, software testing Zend Virtual Machine PHP Code Opcache JIT Buffer JIT Compiler Machine Code