Lune

CVPR Findings2026

Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI

Mingjie Li, Edward Kim, Yue Zhao, Ehsan Adeli, Kilian M. Pohl

Abstract

Learning a robust Variational Autoencoder (VAE) is a fundamental step for many deep learning applications in medical image analysis, such as MRI synthesizes. Existing brain VAEs predominantly focus on single-modality data (i.e., T1-weighted MRI), overlooking the complementary diagnostic value of other modalities like T2-weighted MRIs. Here, we propose a modality-aware and anatomically grounded 3D vector-quantized VAE (VQ-VAE) for reconstructing multi-modal brain MRIs. Called NeuroQuant, it first learns a shared latent representation across modalities using factorized multi-axis attention, which can capture relationships between distant brain regions. It then employs a dual-stream 3D encoder that explicitly separates the encoding of modality-invariant anatomical structures from modality-dependent appearance. Next, the anatomical encoding is discretized using a shared codebook and combined with modality-specific appearance features via Feature-wise Linear Modulation (FiLM) during the decoding phase. This entire approach is trained using a joint 2D/3D strategy in order to account for the slice-based acquisition of 3D MRI data. Extensive experiments on two multi-modal brain MRI datasets demonstrate that Neuro-Quant achieves superior reconstruction fidelity compared to existing VAEs, enabling a scalable foundation for downstream generative modeling and cross-modal brain image analysis.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • Scaling the Codebook Size of VQ-GAN to 100, 000 with a Utilization Rate of 99% cites High-Resolution Image Synthesis with Latent Diffusion Models
  • An Image is Worth 32 Tokens for Reconstruction and Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on8

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. An Image is Worth 32 Tokens for Reconstruction and GenerationQihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen et al.NeurIPS 2024 · 331 citations
  4. Scaling the Codebook Size of VQ-GAN to 100, 000 with a Utilization Rate of 99%Lei Zhu, Fangyun Wei, Yanye Lu, Dong ChenNeurIPS 2024 · 52 citations
  5. Domain Intersection and Domain DifferenceSagie Benaim, Michael Khaitov, Tomer Galanti, Lior WolfICCV 2019 · 28 citations

Related papers

  1. Virtual Full-stack Scanning of Brain MRI via Imputing Any Quantised CodeYicheng Wu, Tao Song, Zhonghua Wu, Jin Ye et al.CVPR 2026 · 4 citations
  2. VQ3D: Learning a 3D-Aware Generative Model on ImageNetKyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang et al.ICCV 2023 · 33 citations
  3. Associative Variational Auto-Encoder with Distributed Latent Spaces and AssociatorsDae Ung Jo, Byeongju Lee, Jongwon Choi, Haanju Yoo et al.AAAI 2020 · 8 citations
  4. Single-Stage fMRI-to-3D Reconstruction via Viewpoint-Aware Embedding and Hierarchical GuidanceXun Zhang, Weihao Xia, Yulong Liu, Bo Yang et al.AAAI 2026
  5. Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision TransformersShiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu et al.ICCV 2023 · 33 citations
Dusk over the sea between two cliffs drawn in fine vertical lines