OccMesh: Occlusion-aware Multi-user 3D Human Mesh Reconstruction Using mmWave Signals

Nowadays, 3D human mesh reconstruction technology has drawn great attention in the metaverse for building digital humans. As current mainstream solutions, vision-based systems suffer from privacy leakage and depend on lighting conditions. Towards a more privacy-preserving and illumination-robust manner, recent works have exploited radio frequency signals to realize 3D human mesh reconstruction. However, these studies cannot handle occlusion scenarios with proximity or overlap in multi-user scenarios. This paper presents an occlusion-aware 3D human mesh reconstruction system, OccMesh, which uses mmWave signals to estimate body skeletons and reconstruct human meshes of users with proximity or overlap. In this paper, OccMesh first detects subjects and generates point clouds through mmWave signals, and then segments point clouds to identify and locate each user. OccMesh next estimates human body keypoints of multiple users through a designed RBF Value Estimator. When there are multiple users in close proximity or overlapped, OccMesh infers the occluded human body parts and generates the skeletons of occluded users with a ResGCN-Attention-Block-based Skeleton Inferer. Based on the generated skeletons, OccMesh further reconstructs human meshes for users in occlusion scenarios. Experiments conducted in occlusion scenarios validate the accuracy and robustness of OccMesh in multi-user 3D human mesh reconstruction.