Demystifying the Access Control Mechanism of ESXi VMKernel

VMware ESXi is a widely deployed enterprise-grade Type-1 hypervisor that serves as the foundation for modern cloud infrastructure. To reinforce privilege isolation, ESXi introduced a mandatory access control mechanism in VMKernel. However, due to VMKernel's proprietary and closed-source nature, its internal access control architecture remains largely opaque and underexplored. Prior research has focused primarily on virtual device vulnerabilities and virtual machine escape, leaving the internal access control mechanisms and privilege model of VMKernel largely unexamined. To address this gap, we conduct the first comprehensive security analysis of VMKernel's access control mechanism. We develop a domain-control structure oriented analysis method to reconstruct key internal permission logic, and design a structureaware debugging framework to support fine-grained runtime validation. Using this framework, we uncover several critical design flaws, including writable and unprotected in-memory control structures and exploitable developer-reserved syscall interfaces. We demonstrate three practical attack scenarios that abuse these flaws to bypass sandbox restrictions, escalate privileges, and gain persistent access. In total, we discovered and reported 14 vulnerabilities to VMware, all of which have been confirmed and fixed, with a total of $42,000 in bug bounties awarded. I. INTRODUCTION VMware ESXi is a leading enterprise-grade Type-1 virtualization platform, widely deployed in private clouds, enterprise data centers, and other mission-critical environments. It is built on a bare-metal architecture that delivers high performance, strong isolation, and fine-grained resource scheduling, enabling large-scale virtual machine(VM) deployments and high-availability cluster management. Within the bare-metal hypervisor segment, ESXi holds over 45% of the global market share [1], establishing itself as a cornerstone of modern cloud infrastructure and playing a critical role in business continuity, security, and elastic resource management.