mmMulti: Multi-person Action Recognition Based on Multi-task Learning Using Millimeter Waves

Action recognition using millimeter waves (mmWave) has shown great potential in various fields. However, the state-of-the-art still falls short in terms of multi-person action recognition inclusive interactive action recognition. In this paper, we propose mmMulti, a multi-person action recognition method based on multi-task learning using millimeter waves. To this end, we first segregate the mmWave data and assign them to each of multiple persons, and propose two new input data---compressed Doppler map (CDM) and point trajectory segments (PTS)---to represent the patterns and sequential characteristics of actions. Next, we leverage ConvNeXt to extract pattern features from CDM and leverage Transformer to extract sequential features from PTS, and fuse them by a cross-attention mechanism. Finally, we custom-design a multi-task learning model to recognize independent and interactive actions from multiple concurrent persons, enabling mmMulti to recognize single-person actions, multi-person independent actions and multi-person interactive actions. We implement mmMulti on a commercial mmWave radar and conduct extensive experiments. mmMulti achieves single-person action recognition accuracy of 99.64%, independent action recognition accuracy of 91.03% for two persons, 72.38% for three persons, 64.75% for four persons, and interactive action recognition accuracy of 100% for two persons. To the best of our knowledge, mmMulti is the first work in the field of mmWave sensing to differentiate both independent and interactive actions in multi-person scenarios, based on a multi-task learning model to accomplish multiple tasks simultaneously.