Ubi Grip: Ubiquitous Grip-Based Tangible Object Utilization in Augmented Reality

Tangible Augmented Reality (AR) enhances user immersion in virtual world by providing haptic feedback through physical proxy objects. However, existing approaches primarily focus on selecting proxy objects based on their global physical properties, neglecting the utilization of local features. Besides, the prevailing strategy of mapping one virtual object to a single dedicated physical proxy creates an inherent switching cost, limiting flexibility and efficiency. Additionally, due to challenges such as real-time performance, generalization and occlusion, the vision-based hand-object tracking remains a difficult task. In this paper, we propose Ubi Grip, an universal hand-object interaction framework for creating grip-based tangible AR applications based on the local graspable feature and a comprehensive hand-object interaction attributes methodology. We employ a lightweight object tracking method to perform tracking, utilizing a hand mask filter and transformation strategy to optimize object pose based on the hand-held properties. Moreover, we design a user-defined workflow for grasping tangible objects, allowing users to switch grips and map interactions. We evaluated our system through comprehensive algorithmic benchmarks and a user study. The benchmarks demonstrate our SOTA performance in object pose estimation and generalization, while the user study validates system usability, providing deeper insights.