MMHead: Towards Fine-grained Multi-modal 3D Facial Animation

The portrait talks about something. <emotion> The portrait looks surprised throughout the video clip.

The portrait in the video clip starts to frown, then sneers, wags its head, and finally starts talking. <emotion> The portrait's expression remains contemptuous throughout the video clip.

<expression> The subject begins with a strong inner brow raise and outer brow raise, with the lips parting widely and the jaw dropping slightly. As the sequence progresses, the inner brow raise remains consistent, . . . Towards the end of the sequence, the subject displays a mixture of facial movements, including a mouth stretch and tongue show. <head pose> The portrait maintains a steady head pose throughout the video, with no noticeable movements or rotations detected. <scenarios> 1. Finding out he won the lottery; 2. Witnessing a loved one's unexpected return; 3. ... <expression> The lips initially part with a high intensity, followed by the nostrils compressing slightly. ... As time progresses, the lid tightens, the lips continue to part, and the inner brows maintain a high level of intensity ...The nostrils compress periodically, and the cheek on the left side raises slightly. <head pose> The portrait turns its head to the right ... The head then switches direction, turning left gradually while also tilting slightly to the right. ... The head keeps moving upwards, turning left consistently at an increasing angle. Finally, the head settles in an upturned position with a slight leftward turn remaining. <scenarios> 1. The portrait is scrolling through social media and seeing posts from people he Dislike; 2. The portrait is watching a political debate where their least favorite candidate is speaking; 3. ... Figure 1: We present MMHead, the first multi-modal 3D facial animation dataset with hierarchical text annotations including abstract descriptions for overall actions and emotions, and fine-grained descriptions for expressions, head poses, as well as possible scenarios that may cause such emotions.