Emotion Embodied: Unveiling the Expressive Potential of Single-Hand Gestures

Hand gestures are widely used in daily life for expressing emotions, yet gesture input is not part of existing emotion tracking systems. To seek a practical and effortless way of using gestures to inform emotions, we explore the relationships between gestural features and commonly experienced emotions by focusing on single-hand gestures that are easy to perform and capture. First, we collected 756 gestures (in photo and video pairs) from 63 participants who expressed different emotions in a survey, and then interviewed 11 of them to understand their gesture-forming rationales. We found that the valence and arousal level of the expressed emotions significantly correlated with participants’ finger-pointing direction and their gesture strength, and synthesized four channels through which participants externalized their expressions with gestures. Reflecting on the findings, we discuss how emotions can be characterized and contextualized with gestural cues and implications for designing multimodal emotion tracking systems and beyond.