EmoShortcuts: Emotionally Expressive Body Augmentation for Social Mixed Reality Avatars

We present EmoShortcuts1, a novel social Mixed Reality (MR) framework that enhances emotional expression by dynamically augmenting avatar body gestures to reflect users’ emotional states. While social MR enables immersive remote interactions through avatars, conveying emotions remains challenging due to limitations in head-mounted display (HMD) tracking (e.g., missing lower-body movements, such as stomping or defensive postures), and users’ tendency to deprioritize nonverbal expressions during multitasking. EmoShortcuts addresses these challenges by introducing an augmentation framework that generates expressive body gestures even when users’ physical movements are restricted. We conducted a formative study with 12 participants to identify key challenges in emotional expression and explore user preferences for AI-assisted gesture augmentation. Based on these insights, we designed an interface that enables adaptive gesture augmentation, allowing for both preset and real-time user control. Through an extensive user study (n = 16), our findings show that EmoShortcuts significantly improves emotion expression accuracy, controllability, and workload, demonstrating its potential for more immersive and emotionally rich virtual communication.