TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision

Figure 1: We present a novel method for detecting touch input on surfaces from egocentric hand tracking alone. (a) Here, a user is interacting in mixed reality through touch on a desk, which provides haptic feedback. (b) Our method senses input events from egocentric views-a challenging task, as hand self-occlusion causes sensing uncertainty about exact touch locations. Our learning-based method explicitly models these uncertainties and resolves them as part of a probabilistic framework, accounting for user behavior and context to enable rapid and dexterous text input on a virtual surface keyboard (c).