Tiny is not small enough: High quality, low-resource facial animation models through hybrid knowledge distillation
Zhen Han, Mattias Teye, Derek Yadgaroff, Judith Bütepage
Abstract
The training of high-quality, robust machine learning models for speech-driven 3D facial animation requires a large, diverse dataset of high-quality audio-animation pairs. To overcome the lack of such a dataset, recent work has introduced large pre-trained speech encoders that are robust to variations in the input audio and, therefore, enable the facial animation model to generalize across speakers, audio quality, and languages. However, the resulting facial animation models are prohibitively large and lend themselves only to offline inference on a dedicated machine. In this work, we explore on-device, real-time facial animation models in the context of game development. We overcome the lack of large datasets by using hybrid knowledge distillation with pseudo-labeling. Given a large audio dataset, we employ a high-performing teacher model to train very small student models. In contrast to the pre-trained speech encoders, our student models only consist of convolutional and fully-connected layers, removing the need for attention context or recurrent updates. In our experiments, we demonstrate that we can reduce the memory footprint to up to 3.4 MB and required future audio context to up to 81 ms while maintaining high-quality animations. This paves the way for on-device inference, an important step towards realistic, model-driven digital characters.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Imitator: Personalized Speech-driven 3D Facial Animation cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- Imitator: Personalized Speech-driven 3D Facial Animation cites FaceFormer: Speech-Driven 3D Facial Animation with Transformers
- FaceFormer: Speech-Driven 3D Facial Animation with Transformers cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Builds on11
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
- Improved Knowledge Distillation via Teacher AssistantSeyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine et al.AAAI 2020 · 1,361 citations
- FaceFormer: Speech-Driven 3D Facial Animation with TransformersYingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang et al.CVPR 2022 · 218 citations
- Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait SynthesisJiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou et al.ICCV 2023 · 127 citations
- Imitator: Personalized Speech-driven 3D Facial AnimationBalamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliakbarian, Darren Cosker et al.ICCV 2023 · 98 citations
Related papers
- Speech Driven Tongue AnimationSalvador Medina, Denis Tomè, Carsten Stoll, Mark Tiede et al.CVPR 2022 · 14 citations
- StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion ModelYifan Yang, Zhi Cen, Sida Peng, Xiangwei Chen et al.AAAI 2026 · 1 citation
- CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion PriorJinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun et al.CVPR 2023
- Semi-supervised Speech-driven 3D Facial Animation via Cross-modal EncodingPeiji Yang, Huawei Wei, Yicheng Zhong, Zhisheng WangICCV 2023 · 1 citation
- Media2Face: Co-speech Facial Animation Generation With Multi-Modality GuidanceQingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin et al.SIGGRAPH 2024 · 40 citations
