IsolatOS: Detecting Double Fetch Bugs in COTS RTOS by Re-enabling Kernel Isolation

—Real-time operating systems (RTOS) often expose double-fetch vulnerabilities when the kernel reads the same user-space memory location multiple times without ensuring consistency between fetches. Conventional static analysis cannot inspect proprietary, commercial off-the-shelf (COTS) RTOS kernels, and dynamic heuristics, which rely on broad time-window thresholds, suffer from high false positive rates and heavy emulation overhead. To address these challenges, we present ISOLAT OS, the first hardware-supported framework for detecting double-fetch bugs in COTS RTOS. By leveraging modern CPU kernel-isolation features, ISOLAT OS enables kernel isolation so that cross-boundary accesses can be captured by triggering page faults. ISOLAT OS then records page-fault metadata on each user-memory fetch. Finally, multiple fetches in the same system call are determined as a double-fetch bug, based on the lifecycle of system calls that ISOLAT OS instruments into COTS RTOS. We evaluate ISOLAT OS on three widely used RTOS, including QNX, VxWorks, and seL4, and demonstrate a 79.3 × reduction in runtime overhead compared to state-of-the-art emulation-based detectors. ISOLAT OS also detects double-fetch bugs with lower false positive rates than other tools. Our approach uncovers 43 previously unknown vulnerabilities in COTS RTOS (41 confirmed by vendors, 2 CVEs assigned). Additionally, we have demonstrated the real-world impact of our findings in automotive systems by exploiting them.