MagSkin: Enabling Touchless Human-Computer Interaction with Soft Epidermal Magnetic Material

Soft epidermal magnetic material has demonstrated excellent flexibility and wearability across a range of emerging applications, e.g., pressure sensing, robotics, among others. In this paper, we build a flexible magnetic skin based system MagSkin for touchless Human Computer Interaction (HCI). For example, a user can wear a Bandaid-like sticker on his fingertip to operate a smartphone using classic gestures without touching the screen, i.e., tap virtual icons floating on top of the screen. The magnetic skin is specially fabricated with randomly distributed magnetic microparticles, so that any shape change of the skin will cause position and orientation changes of those particles, introducing complex variations of the magnetic field around the skin. To characterize the magnetic field of the magnetic skin, we designed a multi-dipole based model. Based on this model, we developed 6-DoF position tracking and orientation estimation algorithms to localize the skin precisely. We also designed novel gestures, e.g., finger bend and pinch, for users to make input/adjustment based on the flexibility of the material. We have conducted experiments with 15 volunteers for 18 hours, experimental results show localization accuracy is 6.2 mm in 3D, 4.5 mm in 2D, and recognition accuracy is 94% for classical gesture and 92% for novel gesture.