Building Dynamic System Call Sandbox with Partial Order Analysis

Attack surface reduction is a security technique that secures the operating system by removing the unnecessary code or features of a program. By restricting the system calls that programs can use, the system call sandbox is able to reduce the exposed attack surface of the operating system and prevent attackers from damaging it through vulnerable programs. Ideally, programs should only retain access to system calls they require for normal execution. Many researchers focus on adopting static analysis to automatically restrict the system calls for each program. However, these methods do not adjust the restriction policy along with program execution. Thus, they need to permit all system calls required for program functionalities.

We observe that some system calls, especially security-sensitive ones, are used a few times in certain stages of a program's execution and then never used again. This motivates us to minimize the set of required system calls dynamically. In this paper, we propose DynBox, which gradually disables access to unnecessary system calls throughout the program's execution. To accomplish this, we utilize partial order analysis to transform the program into a partially ordered graph, which enables efficient identification of the necessary system calls at any given point during program execution. Once a system call is no longer required by the program, DynBox can restrict it immediately. To evaluate DynBox, we applied it to seven widely-used programs with an average of 615 KLOC, including web servers and databases. With partial order analysis, DynBox restricts an average of 23.50, 16.86, and 15.89 more system calls than the state-of-the-art Chestnut, Temporal Specialization, and the configuration-aware sandbox, C2C, respectively. For mitigating malicious exploitations, on average, DynBox defeats 83.42% of 1726 exploitation payloads with only a 5.07% overhead.