HW-Spy: Handwriting Inference by Tracing Pen-Tail Movements

While keyboard typing has been the most common way of inputting texts, handwriting still plays an important role in generating, inputting, or recording information like filling out essential/private forms. Considerable research has been done to identify and demonstrate the risk of keystroke-inference attacks. However, little has been done on handwriting inference despite its high risk of leaking sensitive information. To assess this under-explored risk of information leakage, we present a novel handwriting-inference attack, called HW-Spy, by tracing the victim's pen-tail movements when both the pen tip and the writing surface are outside the view of the attacker's camera, which usually happens when the victim is multitasking during an online meeting, when the victim's writing scene (in a public space) is recorded by a remote camera, or when the victim's writing behaviors are captured by the surveillance camera in a bank/dealership/realty office.

In particular, we apply image segmentation to the recorded video frames of the victim's writing activities and extract the victim's pentail movements as a 2D coordinate sequence. We then identify the stroke-associated movements from the recorded pen's in-air video frames using a 1D U-Net model trained for stroke mask prediction and segment the characters based on the thus-derived motion features. The pen-tail's coordinate segments are then fed into a Long Short-Term Memory (LSTM) network to reconstruct the actual handwriting, which is processed further by a transformer-based model to infer the hand-written content. Our extensive experimentation shows HW-Spy to achieve an accuracy, up to 84.2%, of personalized handwriting inference and a comparable accuracy, up to 79.5%, of non-personalized handwriting inference.