Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling

Listening head generation aims to synthesize non-verbal responsive listening head videos that naturally react to a certain speaker, for which, both realistic head movements, expressive facial expressions, and high visual qualities are expected. Previous approaches typically follow a two-stage pipeline that first generates intermediate 3D motion signals such as 3DMM coefficients, and then synthesizes the videos by deterministic rendering, suffering from limited motion expressiveness and low visual quality (e.g. 256 × 256). In this work, we propose a novel listening head generation method that harnesses the generative capabilities of the diffusion model for both motion generation and high-quality rendering. Crucially, we propose an effective hybrid motion modeling module that addresses training difficulties caused by the scarcity of listening head data while preserving the intricate details that may be lost in explicit motion representations. We further develop a tailored control guidance for head pose and facial expression, by integrating their intrinsic motion characteristics. Our method enables highfidelity video generation with 512 × 512 resolution and delivers vivid listener motion feedback. We conduct comprehensive experiments and obtain superior performance in terms of both visual quality and motion expressiveness compared with existing methods. * Equal contribution † Corresponding authors ‡ This work was done during Yinuo Wang's internship at Ant Group ing, frowning, head shaking, etc. The realism and vividness of the listener feedback are vital to the success of speaker and listener communication. Being a crucial part of digital avatar generation, listening head generation has many potential applications, such as movie production, online customer service, and human-computer interaction.