Human Parsing with Joint Learning for Dynamic mmWave Radar Point Cloud

Human sensing and understanding is a key requirement for many intelligent systems, such as smart monitoring, humancomputer interaction, and activity analysis, etc. In this paper, we present mmParse, the first human parsing design for dynamic point cloud from commercial millimeter-wave radar devices. mmParse proposes an end-to-end neural network design that addresses the inherent challenges in parsing mmWave point cloud (e.g., sparsity and specular reflection). First, we design a novel multi-task learning approach, in which an auxiliary task can guide the network to understand human structural features. Secondly, we introduce a multi-task feature fusion method that incorporates both intra-task and inter-task attention to aggregate spatio-temporal features of the subject from a global view. Through extensive experiments in both indoor and outdoor environments, we demonstrate that our proposed system is able to achieve ∼ 92% accuracy and ∼ 84% IoU accuracy. We also show that the predicted semantic labels can increase the performance of two downstream tasks (pose estimation and action recognition) by ∼ 18% and ∼ 6% respectively. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing.