HandCraft: Tactile-Informed Hand-Object Dynamics Capture and Realistic Rendering

Creating digitalized hand-object interaction scenes plays a crucial role in recent advancements, enabling viewers to understand how human dexterity influences and shapes the world. In this paper, we present HandCraft, a framework designed to capture and render hand-object interactions with exceptional precision and realism. Our Gaussian models are built on the development of digital representations of hands, objects, and scenes, derived from data captured using multi-modal sensing systems. By combining motion capture with IMU-based data gloves equipped with tactile sensors, HandCraft ensures precise hand pose tracking and reliable contact fidelity. HandCraft includes a novel method that uses hand motions to solve the object occlusions, effectively reconstructing missing interaction details. For enhanced physical feasibility, HandCraft incorporates optimization techniques to resolve object penetration issues and enforce temporal consistency. Using these techniques, we introduce a high-quality dataset of hand-object interaction sequences, featuring complex and prolonged daily activities. This dataset demonstrates HandCraft's ability to capture and reproduce subtle, dynamic interactions in rich detail. HandCraft holds promises in creating realistic virtual environments and advancing world modeling in both graphics and robotics research.