Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision

Egocentric human body estimation allows for the inference of user body pose and shape from a wearable camera's firstperson perspective. Although pose estimation techniques have been used to overcome self-occlusions and image distortions caused by head-mounted fisheye images, similar advances in 3D human mesh recovery (HMR) techniques have been limited. We address this gap with Fish2Mesh, a fisheye-aware transformer-based model designed for 3D egocentric human mesh recovery. We propose an egocentric position embedding block to generate an ego-specific position table for the Swin Transformer to reduce fisheye image distortion. Our model utilizes multi-task heads for SMPL parametric regression and camera translations, estimating 3D and $2 D$ joints as auxiliary loss to support model training. Further, we augment egocentric camera data with a training dataset by employing the pre-trained 4D-Human model and third-person cameras for weak supervision. Our experiments demonstrate that Fish2Mesh outperforms state-of-the-art 3D HMR models. Code and data are available on our website.