m3ASL: ASL Gesture Recognition with Moving mmWave Radar

This paper tries to answer the question: “Can we enable moving mmWave radar to recognize ASL gestures?” A positive answer would help facilitate the non-contact interaction between the millions of deaf or hard of hearing ASL users and radar-equipped mobile robots. The challenges, however, include how to deal with the pros and cons of radar motion, and how to exploit the multi-scale nature of radar data. This paper gives an affirmative answer by proposing m3ASL, a method which achieves ASL recognition with moving mmWave radar for the first time. Specifically, m3ASL generates accurate radar point cloud in a motion-ware manner: it constructs extended phase sequences for accurate angle estimation by leveraging radar motion, corrects motion-incurred phase errors via self-supervision, and calibrates motion-incurred point cloud distortion via careful spatial transformation and speed compensation. Furthermore, m3ASL extracts features from multi-scale radar data accordingly with augmented graph and temporal convolution neural network, and complementarily fuses the extracted features with our proposed symmetric double cross attention neural network structure to infer ASL gestures. We conduct extensive experiments via both simulation and real-world system. The experimental results show that m3ASL achieves close to 93% recognition accuracy on 30 representative ASL gestures, surpassing SOTA by at least 30%.