Eyes on your Typing: Snooping Finger Motions on Virtual Keyboards

The rapid growth of augmented reality (AR) and virtual reality (VR) technologies has introduced immersive digital experiences for consumers across numerous fields, including banking, education, and professional spheres. In these environments, head-mounted displays (HMDs) enable users to interact with virtual objects through head and hand tracking. In particular, virtual keyboards are emerging as a primary input method, allowing users to type directly with their hands-eliminating the need for additional devices and adding convenience for portable HMD use. However, this direct hand-based typing introduces new security concerns, namely subtle head movements that occur during direct hand-based typing can unintentionally reveal private information. In this paper, we propose SNOOPFINGER, a novel side-channel attack that leverages head movement data, which is accessible without additional user permissions, to estimate typed inputs on a virtual keyboard. Unlike previous methods, SNOOPFINGER uniquely employs a cross-modality approach, relying solely on head movement data to infer hand-typed inputs without the use of controllers. Additionally, our approach is designed to identify a victim's typed inputs without requiring prior access to extensive head movement data from the victim or other users. In an experiment involving 24 participants, SNOOPFING ER achieved high inference accuracy rates, with an average Top-1 accuracy of 55.2% for word inference and 68.8% for sentence reconstruction. Finally, we discuss potential mitigation strategies to counteract such attacks. Our findings reveal critical privacy risks associated with direct hand-based typing in AR/VR environments, demonstrating how zero-permission sensor data can be exploited to obtain private information.