REMEMBER: Retrieval-based Explainable Multimodal Evidence-guided Modeling for Brain Evaluation and Reasoning in Zero- and Few-shot Neurodegenerative Diagnosis

Timely and accurate diagnosis of neurodegenerative disorders, such as Alzheimer's disease, is central to disease management. Existing deep learning models require large annotated datasets and often act as ''black boxes''. However, clinical datasets are frequently small or lack labels, limiting the effectiveness of these methods. Here, we introduce REMEMBER - Retrieval-based Explainable Multimodal Evidence-guided Modeling for Brain Evaluation and Reasoning - a machine learning framework that enables zero- and few-shot Alzheimer's diagnosis from brain MRI scans via reference-based reasoning. Specifically, REMEMBER first contrastively trains a vision-text model on expert-annotated reference data, using pseudo-text modalities to encode abnormality types, diagnosis labels, and composite clinical descriptions. At inference time, it retrieves similar, human-validated cases from a curated dataset and integrates their contextual information via an evidence encoder and attention-based inference head. This evidence-guided design allows REMEMBER to mimic clinical decision-making by grounding predictions in retrieved imaging and textual context. It outputs diagnostic predictions with an interpretable report, including reference images and clinical-aligned explanations. Experimental results demonstrate that REMEMBER achieves robust zero- and few-shot performance and offers a powerful and explainable framework to neuroimaging-based diagnosis in the real world, especially under limited data.