LAM-assisted Acoustic Eavesdropping in Multi-speaker Scenarios via Commercial mmWave Radar

Eavesdropping on target speech in enclosed and private rooms, such as secret gatherings or private parties, has become a growing research trend and poses one of the most significant threats to personal privacy leakage. Previous studies often made various assumptions about the target and scenarios of eavesdropping, such as the presence of only a single speaker, a very slow speech rate, and clear separation of words, all of which greatly limited the applicability of eavesdropping systems. In this paper, we propose an acoustic eavesdropping system, mmMPS, based on commercial off-the-shelf millimeter-wave (COTS mmWave) radar, which is capable of eavesdropping on specific target speech in scenarios where multiple people speak simultaneously, without requiring prior knowledge of background voices, speech rate, or intensity. The system utilizes a single COTS mmWave radar to capture faint signals caused by the vibration of objects due to human speech, combines these with our proposed denoising algorithm to recover the speech signal, and then uses our recognition model along with large audio model (LAM) to extract target keyword speech from the mixed speech signals of multiple speakers. Extensive experiments were conducted on a nearly 120 hours audio dataset and 60 hours radar dataset that we created using LAM and collected in experimental scenarios. The system is capable of covering various target users, speech rates, and speech intensities. The results show that in a scenario with 1–2 simultaneous speakers, the system achieves the average WER/CER (Word/Character Error Rate) of 6.46% (the lower, the better) in identifying 36 target keywords. In a scenario of 3–5 speakers, the average WER/CER is 14.97%, and even in the scenario with up to 7 simultaneous speakers, the average WER/CER remains below 30%. To the best of our knowledge, mmMPS is the first system to achieve high-accuracy eavesdropping on target speech in multi-speaker scenarios, significantly expanding the application scope of acoustic eavesdropping.