HandSAW: Wearable Hand-based Event Recognition via On-Body Surface Acoustic Waves

Enabling computing systems to detect the objects that people hold and interact with provides valuable contextual information that has the potential to support a wide variety of mobile applications. However, existing approaches either directly instrument users' hands, which can reduce tactile sensation, or are limited in the types of objects and interactions they can detect. This work introduces HandSAW, a wireless wrist-worn device incorporating a Surface Acoustic Wave (SAW) sensor with enhanced bandwidth and signal-to-noise ratio while rejecting through-air sounds. The device features a sealed mass-spring diaphragm positioned on top of the sound port of a MEMS microphone, enabling it to capture SAWs generated by objects and through touch interaction events. This custom-designed wearable platform, paired with a real-time ML pipeline, can distinguish 20 passive object events with >99% per-user accuracy and a 91.6% unseen-user accuracy, as validated through a 16-participant user study. For devices that do not emit SAWs, our active tags enable HandSAW to detect those objects and transmit encoded data using ultrasonic signals. Ultimately, HandSAW provides an easy-to-implement, robust, and cost-effective means for enabling user-object interaction and activity detection.