The Motion is the Message: Evaluating Motion Tracking Quality for VR Avatars

Motion tracking to project users into embodied virtual reality (VR) as avatars is an essential application of real-time computer graphics. Most current embodied VR systems rely on head-mounted displays (HMDs) to estimate user pose, as headset sensors can track the head and hands, thereby reconstructing the full body without the need for external hardware. However, measuring the quality of motion reconstruction algorithms from HMD-based tracking, particularly those intended for use in social settings, remains challenging due to the complex interaction between motion and perceived social signals. This paper compares two industrial tracking reconstruction solutions, called HMD1 (i.e., a basic HMD-based method that uses head tracking and hand positions estimated from HMD cameras) and HMD2 (i.e., an advanced HMD-based method with additional onboard camera streams), that estimate user motion using only an HMD against ground-truth motion capture (MoCap) data. It advocates for a social signal-based analysis that views motion as a communication medium and employs user observations to measure whether viewers successfully perceive the information encoded in motion. Across 156 socially expressive clips, Social Signal ratings were more effective than generic measures at revealing differences between the HMD methods. HMD2 preserved social meaning more accurately than HMD1, with fewer significant deviations from MoCap, while both HMD methods were frequently rated less natural than MoCap. A qualitative review localized recurrent failure modes, such as arm swivel/shoulder errors, posture reconstruction issues, and floating/stance artifacts, which help explain the misreading of social signals. We release a dashboard scorecard, motion capture data, and a benchmark protocol to enable consistent motion evaluation. More generally, this work advocates for an underexplored approach to motion evaluation that focuses on assessing the semantics of motion to determine quality. As reliance on generative artificial intelligence (AI) increases, it is essential to standardize evaluation to preserve the authenticity of the social signals conveyed. The developed dataset and the evaluation framework are provided on our project's website: https://github.com/facebookresearch/MotionIsTheMessageDataset.