CAFI-AR: Contact-aware Freehand Interaction with AR Objects

Freehand interaction enhances user experience, allowing one to use bare hands to manipulate virtual objects in AR. Yet, it remains challenging to accurately and efficiently detect contacts between real hand and virtual object, due to the imprecise captured/estimated hand geometry. This paper presents CAFI-AR, a new approach for Contact-Aware Freehand Interaction with virtual AR objects, enabling us to automatically detect hand-object contacts in real-time with low latency. Specifically, we formulate a compact deep architecture to efficiently learn to predict hand action and contact moment from sequences of captured RGB images relative to the 3D virtual object. To train the architecture for detecting contacts on AR objects, we build a new dataset with 4,008 frame sequences, each with annotated hand-object interaction information. Further, we integrate CAFI-AR into our prototyping AR system and develop various interactive scenarios, demonstrating fine-grained contact-aware interactions on a rich variety of virtual AR objects, which cannot be achieved by existing AR interaction approaches. Lastly, we also evaluate CAFI-AR, quantitatively and qualitatively, through two user studies to demonstrate its effectiveness in terms of accurately detecting the hand-object contacts and promoting fluid freehand interactions