EEGMirror: Leveraging EEG Data in the Wild Via Montage-Agnostic Self-Supervision for EEG to Video Decoding

Generating high fidelity video from brain activity is an important milestone in brain decoding research. Previous works were mostly based on functional Magnetic Resonance Imaging (fMRI), whose low temporal resolution confines the ability of faithfully reflecting rapid brain activity, motivating us to turn to high temporal resolution brain signals like electroencephalography (EEG). However, EEGto-video is challenging due to the complexity and nonstationarity of EEG signals and the scarcity of annotated data. Addressing these issues, we present EEGMirror. Firstly, we adopt neural quantization to convert nonstationary signals into robust discrete representation. Afterwards, a masked self-supervision method with montage-agnostic position embedding (MAPE) is introduced to acquire an effective EEG encoder. By MAPE, our model can flexibly leverage different EEG datasets with various montages (number and position of channels), mitigating the lack of wellannotated data. Next, multimodal contrastive learning is applied to align the brain modality with dynamic changes and semantic information. Lastly, a fine-tuned inflated Stable Diffusion model is adopted to reconstruct video stimuli guided by visual and semantic information decoded from EEG signals. We show that EEGMirror outperforms the state-of-the-art performance in both semantic (82.1% vs 79.8%) and pixel (0.261 vs 0.256) levels. An exhaustive ablation study is also conducted to analyze our framework. Code: https://github.com/XuanhaoLiu/EEGMirror.