Fractal: An Operating System Designed for Microarchitecture Reverse Engineering

Modern microarchitecture security research requires a deep understanding of the microarchitecture designs of commodity hardware, specifically how the hardware is isolated between privilege domains. Since these details are usually undocumented, researchers must conduct reverse engineering experiments to learn the microarchitecture parameters of various processors. Currently, the state of the art approach is to modify commodity operating systems in an ad-hoc manner to enable these experiments. Our objective is to systematize the requirements of microarchitecture security research workflows, and to create new lightweight system software precisely tailored to these requirements. We present Fractal, a new operating system kernel built from the ground up to enable practical low-noise microarchitecture reverse engineering research at the user to kernel hardware boundary. Fractal enables seamless concurrency between privilege levels and a user-controlled scheduler for fine-grained thread ordering to enable new microarchitecture reverse engineering workflows with minimal noise. We ported Fractal to a variety of real systems and evaluate Fractal by using it to analyze the Apple M1 CPU. We uncovered a number of new insights about the branch predictor on M1, demonstrating for the first time evidence that limited Phantom speculation is present on Apple Silicon.