mmWave-Aided Unified Speech Enhancement and Separation without Speaker Count Prior

Automatic speech enhancement and separation are critical for improving speech quality and intelligibility in noisy, multi-speaker environments, with applications in hearing aids, automatic speech recognition (ASR), and voice-controlled systems. However, most existing methods rely on task-specific models designed for a fixed number of speakers—such as one model for single-speaker enhancement and others for two- or three-speaker separation—requiring explicit prior knowledge of speaker count. This design paradigm limits flexibility and scalability in real-world scenarios where the number of speakers is often unknown and dynamic. To overcome this limitation, we introduce Ra-dioSEP, a unified mmWave-audio multimodal framework that performs both speech enhancement and separation without prior knowledge of speaker count. By leveraging mmWave radar sensing, RadioSEP detects, localizes, and profiles multiple speakers, enabling dynamic estimation of speaker count and extraction of speaker-specific physical features. These features are fused with noisy audio and processed by a single deep neural network that adapts to the current speaker configuration, generating clean speech streams accordingly. Extensive experiments show that RadioSEP consistently outperforms state-of-the-art methods in both speech enhancement and multi-speaker separation tasks, while offering significantly improved generalization and adaptability.