CVPR Findings2026
Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI
Mingjie Li, Edward Kim, Yue Zhao, Ehsan Adeli, Kilian M. Pohl
Abstract
Learning a robust Variational Autoencoder (VAE) is a fundamental step for many deep learning applications in medical image analysis, such as MRI synthesizes. Existing brain VAEs predominantly focus on single-modality data (i.e., T1-weighted MRI), overlooking the complementary diagnostic value of other modalities like T2-weighted MRIs. Here, we propose a modality-aware and anatomically grounded 3D vector-quantized VAE (VQ-VAE) for reconstructing multi-modal brain MRIs. Called NeuroQuant, it first learns a shared latent representation across modalities using factorized multi-axis attention, which can capture relationships between distant brain regions. It then employs a dual-stream 3D encoder that explicitly separates the encoding of modality-invariant anatomical structures from modality-dependent appearance. Next, the anatomical encoding is discretized using a shared codebook and combined with modality-specific appearance features via Feature-wise Linear Modulation (FiLM) during the decoding phase. This entire approach is trained using a joint 2D/3D strategy in order to account for the slice-based acquisition of 3D MRI data. Extensive experiments on two multi-modal brain MRI datasets demonstrate that Neuro-Quant achieves superior reconstruction fidelity compared to existing VAEs, enabling a scalable foundation for downstream generative modeling and cross-modal brain image analysis.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
- Scaling the Codebook Size of VQ-GAN to 100, 000 with a Utilization Rate of 99% cites High-Resolution Image Synthesis with Latent Diffusion Models
- An Image is Worth 32 Tokens for Reconstruction and Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
Builds on8
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- An Image is Worth 32 Tokens for Reconstruction and GenerationQihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen et al.NeurIPS 2024 · 331 citations
- Scaling the Codebook Size of VQ-GAN to 100, 000 with a Utilization Rate of 99%Lei Zhu, Fangyun Wei, Yanye Lu, Dong ChenNeurIPS 2024 · 52 citations
- Domain Intersection and Domain DifferenceSagie Benaim, Michael Khaitov, Tomer Galanti, Lior WolfICCV 2019 · 28 citations
Related papers
- Virtual Full-stack Scanning of Brain MRI via Imputing Any Quantised CodeYicheng Wu, Tao Song, Zhonghua Wu, Jin Ye et al.CVPR 2026 · 4 citations
- VQ3D: Learning a 3D-Aware Generative Model on ImageNetKyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang et al.ICCV 2023 · 33 citations
- Associative Variational Auto-Encoder with Distributed Latent Spaces and AssociatorsDae Ung Jo, Byeongju Lee, Jongwon Choi, Haanju Yoo et al.AAAI 2020 · 8 citations
- Single-Stage fMRI-to-3D Reconstruction via Viewpoint-Aware Embedding and Hierarchical GuidanceXun Zhang, Weihao Xia, Yulong Liu, Bo Yang et al.AAAI 2026
- Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision TransformersShiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu et al.ICCV 2023 · 33 citations
