EMOPortraits: Emotion-Enhanced Multimodal One-Shot Head Avatars
Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, Maja Pantic
Abstract
Abstract Head avatars animated by visual signals have gained popularity, particularly in cross-driving synthesis where the driver differs from the animated character, a challenging but highly practical approach. The recently presented MegaPortraits model has demonstrated state-of-the-art results in this domain. We conduct a deep examination and evaluation of this model, with a particular focus on its latent space for facial expression descriptors, and uncover several limitations with its ability to express intense face motions. To address these limitations, we propose substantial changes in both training pipeline and model architecture, to introduce our EMOPortraits model, where we:
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
17 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- FLOAT: Generative Motion Latent Flow Matching for Audio-Driven Talking Portrait cites VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time
- FLOAT: Generative Motion Latent Flow Matching for Audio-Driven Talking Portrait cites A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
- HeadGAN: One-shot Neural Head Synthesis and Editing cites Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking cites FLOAT: Generative Motion Latent Flow Matching for Audio-Driven Talking Portrait
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking cites VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking cites Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking cites A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time cites A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
- Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis cites VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time
- Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis cites A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
- I M Avatar: Implicit Morphable Head Avatars from Videos cites Neural Head Avatars from Monocular RGB Videos
Cited by top-tier papers17
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real TimeSicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang et al.NeurIPS 2024 · 253 citations
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style MimickingZhongjian Wang, Peng Zhang, Jinwei Qi, Yuan Wang et al.NeurIPS 2025 · 12 citations
- MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion TransformerShurong Yang, Huadong Li, Juhao Wu, Minhao Jing et al.AAAI 2025 · 12 citations
- FLOAT: Generative Motion Latent Flow Matching for Audio-Driven Talking PortraitTaekyung Ki, Dongchan Min, Gyeongsu ChaeICCV 2025 · 6 citations
- Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head SynthesisTianqi Li, Ruobing Zheng, Minghui Yang, Jingdong Chen et al.ACM MM 2025 · 4 citations
Builds on17
- A Lip Sync Expert Is All You Need for Speech to Lip Generation In the WildK. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C. V. JawaharACM MM 2020 · 869 citations
- Few-Shot Adversarial Learning of Realistic Neural Talking Head ModelsEgor Zakharov, Aliaksandra Shysheya, Egor Burkov, Victor S. LempitskyICCV 2019 · 687 citations
- Neural Head Avatars from Monocular RGB VideosPhilip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother et al.CVPR 2022 · 173 citations
- I M Avatar: Implicit Morphable Head Avatars from VideosYufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen et al.CVPR 2022 · 169 citations
- HeadGAN: One-shot Neural Head Synthesis and EditingMichail Christos Doukas, Stefanos Zafeiriou, Viktoriia SharmanskaICCV 2021 · 164 citations
Related papers
- MegaPortraits: One-shot Megapixel Neural Head AvatarsNikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko et al.ACM MM 2022 · 86 citations
- DeX-Portrait: Disentangled and Expressive Portrait Animation via Explicit and Latent Motion RepresentationsYuxiang Shi, Zhe Li, Yanwen Wang, Hao Zhu et al.CVPR 2026 · 3 citations
- X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent AttentionXiaochen Zhao, Hongyi Xu, Guoxian Song, You Xie et al.ICLR 2025
- GeoDiff4D: Geometry-Aware Diffusion for 4D Head Avatar ReconstructionChao Xu, Xiaochen Zhao, Xiang Deng, Jingxiang Sun et al.CVPR 2026
- Emotion-Conditioned Motion Sub-spaces with Flow Matching for Real-Time Audio-Driven Talking HeadsHaoyu Wang, Xiaozhe Xin, Xiaoyu Qin, Meiguang Jin et al.AAAI 2026 · 1 citation
