Jenny: Securing Syscalls for PKU-based Memory Isolation Systems

Effective syscall filtering is a key component for withstanding the numerous exploitation techniques and privilege escalation attacks we face today. For example, modern browsers use sandboxing techniques with syscall filtering in order to isolate critical code. Cloud computing heavily uses containers, which virtualize the syscall interface. Recently, cloud providers are switching to in-process containers for performance reasons, calling for better isolation primitives. A new isolation primitive that has the potential to fill this gap is called Protection Keys for Userspace (PKU). Unfortunately, prior research highlights severe deficiencies in how PKU-based systems manage syscalls, questioning their security and practicability.

In this work, we comprehensively investigate syscall filtering for PKU-based memory isolation systems. First, we identify new syscall-based attacks that can break a PKU sandbox. Second, we derive syscall filter rules necessary for protecting PKU domains and show efficient ways of enforcing them. Third, we do a comparative study on different syscall interposition techniques with respect to their suitability for PKU, which allows us to design a secure syscall interposition technique that is both fast and flexible.

We design and prototype Jenny-a PKU-based memory isolation system that provides powerful syscall filtering capabilities in userspace. Jenny supports various interposition techniques (e.g., seccomp and ptrace), and allows for domainspecific syscall filtering in a nested way. Furthermore, it handles asynchronous signals securely. Our evaluation shows a minor performance impact of 0-5% for nginx.

PKU needs, being either insecure or slow. We design a new syscall interposition technique that is both secure and fast.

We present Jenny, the first comprehensive PKU-based inprocess isolation system offering dynamic syscall filtering in userspace. Filters can act on the same thread and also be nested across PKU domains. Jenny comes with filter rules for protecting PKU domains and also supports advanced filters such as file system protection. Jenny further supports different syscall interposition techniques. Moreover, Jenny is the first PKU system that supports secure signal handlers. Finally, we introduce novel multi-domain call gates needed to safeguard the PKU policy register on x86-64. Our evaluation shows a minor performance impact of 0-5% for nginx. Contribution. We make the following contributions:

• We identify previously unknown syscall attacks on PKUbased isolation systems.

• We derive syscall filter rules necessary for protecting PKU-based isolation domains.

• We perform a comparative study of various syscall interposition techniques for their applicability with PKU and derive a new technique that is tailored for PKU.

• We design Jenny-the first comprehensive PKU-based isolation system that supports secure (same-thread) userspace syscall filtering, secure (async.) signal handling, and secure multi-domain PKU call gates for x86-64.

• We prototype and evaluate Jenny under different interposition techniques and filter rules, and open-source it 1 . Outline. Section 2 gives some background. Section 3 raises challenges, analyzes the syscall interface and derives filter rules. Section 4 handles syscall interpositioning. Section 5 presents our design, which Section 6 evaluates. Section 7 and 8 discuss limitations and related work, and we conclude in Section 9.