HoloLogger: Keystroke Inference on Mixed Reality Head Mounted Displays

When using personal computing services in mixed reality (MR) such as online payment and social media, sensitive information and account passwords must be typed in MR. To design secure MR systems and build up user trust, it is imperative to first understand the security threat to the sensitive MR input. Although keystroke inference attacks by analyzing human-computer interaction in videos or via wireless signals have been successful, they require placing extra hardware near the user which is easily noticeable in practice. In this paper, we expose a more dangerous malware-based attack through the vulnerability that no permission is required for accessing MR motion data. We aim to monitor MR headset motion and infer the user input through a benign App. Realizing the attack system requires addressing unique challenges in MR such as six-degree-of-freedom (6DoF) device motion and no explicit motion signal for keystroke identification. To this end, we present HoloLogger, the first malware-based keystroke inference attack system on HoloLens. HoloLogger is empowered by a 6DoF-head-motion-driven key tracking scheme and an air-tap-pattern-based keystroke inference framework. Extensive evaluations with 25 users and 750 inference trials of passwords consisting of 4–8 lowercase English letters demonstrate that HoloLogger successfully achieves a top-5 accuracy of 93%. HoloLogger is also robust in various environments such as different user positions and input categories.