ElaSleepNet: Exploring an Elastic Multimodal Neural Network for Sleep Staging via Temporal and Contextual Consistency Learning

Integrating multimodal learning (ML) with polysomnography (PSG) has emerged as a research hotspot for reliable sleep staging. However, the complexity of these signals and the discomfort associated with wearing multi-lead devices somewhat limit the feasibility of daily and ubiquitous sleep monitoring. Unfortunately, most existing ML paradigms are constrained by consistent and fixed input patterns. When the number of modalities is less than required by the ML framework, it is easy to cause inference bias, resulting in a significant performance degradation. To this end, we propose an elastic multimodal sleep staging network (ElaSleepNet), consisting of multimodal information completion (MIC) and adaptive cross-modal (ACM) interaction. Specifically, MIC maximizes the consistency of multimodal signals on intra-epoch temporal level and inter-epoch contextual level, thereby enhancing the reasoning and completion abilities of the available modalities for the unavailable modalities. Moreover, we introduce learnable parameters and design the ACM attention mechanism, which allows handling multimodal information interaction while maintaining robustness in the absence of certain modalities. Our ElaSleepNet demonstrates its state-of-the-art on three multimodal sleep datasets. Compared with previous methods, ElaSleepNet can achieve better performance with fewer testing modalities, making it flexible for daily monitoring.