EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams

Monocular egocentric 3D human motion capture is a challenging and actively researched problem. Existing methods use synchronously operating visual sensors (e.g. RGB cameras) and often fail under low lighting and fast motions, which can be restricting in many applications involving head-mounted devices. In response to the existing limitations, this paper 1) introduces a new problem, i.e. 3D human motion capture from an egocentric monoc-ular event camera with a fish eye lens, and 2) proposes the first approach to it called EventEgo 3D (EE3D). Event streams have high temporal resolution and provide reliable cues for 3D human motion capture under high-speed hu-man motions and rapidly changing illumination. The pro-posed EE3D framework is specifically tailored for learning with event streams in the LNES representation, enabling high 3D reconstruction accuracy. We also design a proto-type of a mobile head-mounted device with an event cam-era and record a real dataset with event observations and the ground-truth 3D human poses (in addition to the syn-thetic dataset). Our EE3D demonstrates robustness and su-perior 3D accuracy compared to existing solutions across various challenging experiments while supporting real-time 3D pose update rates of 140Hz.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>https://4dqv.mpi-inf.mpg.de/EventEgo3D/