InteractAvatar: Modeling Hand-Face Interaction in Photorealistic Avatars with Deformable Gaussians

With increasing interest in digital avatars coupled with the importance of expressions and gestures in communication, modeling natural avatar behavior remains an important challenge in many industries such as teleconferencing, gaming, and $A R / V R$. Human hands are the primary tool for interacting with the environment and essential for realistic human behavior modeling, yet existing 3D hand and head avatar models often overlook the crucial aspect of hand-body interactions, such as between hand and face. We present InteractAvatar, the first model to faithfully capture the photorealistic appearance of dynamic hand and nonrigid hand-face interactions. Our novel Dynamic Gaussian Hand model, combining template model and 3D Gaussian Splatting as well as a dynamic refinement module, captures pose-dependent change, e.g. the fine wrinkles and complex shadows that occur during articulation. Importantly, our hand-face interaction module models the subtle geometry and appearance dynamics that underlie common gestures. Through experiments of novel view synthesis, selfreenactment, and cross-identity reenactment, we demonstrate that InteractAvatar can reconstruct hand and handface interactions from multi-view videos with high-fidelity details and be animated with novel poses.