Lune

NeurIPS2024Top-tier venue

ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling

Quanwei Yang, Jiazhi Guan, Kaisiyuan Wang, Lingyun Yu, Wenqing Chu, Hang Zhou, ZhiQiang Feng, Haocheng Feng, Errui Ding, Jingdong Wang, Hongtao Xie

Abstract

Although significant progress has been made in human video generation, most previous studies focus on either human facial animation or full-body animation, which cannot be directly applied to produce realistic conversational human videos with frequent hand gestures and various facial movements simultaneously. To address these limitations, we propose a 2D human video generation framework, named ShowMaker, capable of generating high-fidelity half-body conversational videos based on 2D key points via fine-grained diffusion modeling. We leverage dual-stream diffusion models as the backbone of our framework and carefully design two novel components for crucial local regions (i.e., hands and face) that can be easily integrated into our backbone. Specifically, to handle the challenging hand generation caused by sparse motion guidance, we propose a novel Key Point-based Fine-grained Hand Modeling module by amplifying positional information from raw hand key points and constructing a corresponding key point-based codebook. Moreover, to restore richer facial details in generated results, we introduce a Face Recapture module, which extracts facial texture features and global identity features from the aligned human face and integrates them into the diffusion process for face enhancement. Extensive quantitative and qualitative experiments demonstrate the superior visual quality and temporal consistency of our method. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
  • FreeMem: Enhancing Consistency in Long Video Generation via Tuning-Free Memory cites Learning Transferable Visual Models From Natural Language Supervision
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Denoising Diffusion Probabilistic Models
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Denoising Diffusion Implicit Models
  • Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. FreeMem: Enhancing Consistency in Long Video Generation via Tuning-Free MemoryJibin Peng, Di Lin, Zhecheng Xu, Haoran Lu et al.AAAI 2026

Builds on37

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  4. Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
  5. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific TuningYuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang et al.ICLR 2024 · 1,493 citations

Related papers

  1. Make-Your-Anchor: A Diffusion-based 2D Avatar Generation FrameworkZiyao Huang, Fan Tang, Yong Zhang, Xiaodong Cun et al.CVPR 2024
  2. HeroMaker: Human-centric Video Editing with Motion PriorsShiyu Liu, Zibo Zhao, Yihao Zhi, Yiqun Zhao et al.ACM MM 2024
  3. RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated ImagesBenzhi Wang, Jingkai Zhou, Jingqi Bai, Yang Yang et al.AAAI 2025 · 10 citations
  4. StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human AvatarsZhiyao Sun, Ziqiao Peng, Yifeng Ma, Yi Chen et al.CVPR 2026 · 26 citations
  5. 3D²-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar ModelingZichen Tang, Hongyu Yang, Hanchen Zhang, Jiaxin Chen et al.AAAI 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines