Lune

CVPR Findings2026

MeMix: Multi-Encoder Mixture Framework for Medical Report Generation

Yiming Cao, Lizhen Cui, Zhiqi Shen

Abstract

Clinical medical report generation, traditionally performed by doctors, is a time-consuming task. Automating generation can significantly alleviate the workload of medical professionals. However, existing approaches typically leverage various separate multimodal encoders to align image and text features, leading to the failure of effective differentiated fusion of independently encoded medical representations. To address the issues, we propose a multi-encoder mixture framework (MeMix), a novel architecture that leverages the capabilities of multiple encoders to efficiently integrate multimodal medical features for the medical report generation task. Specifically, to exploit the distinct strengths of multiple encoders for multimodal medical data integration, MeMix adopts an image-enhancement unit and a graph-node encoder to derive multi-faceted representations of medical images and domain knowledge. For deep cross-modal semantic fusion, we introduce a novel encoder mixture module based on the Mixture-of-Experts (MoE) architecture, which combines modality-specific and cross-encoder mixtures to capture a comprehensive representation of medical information. Extensive experiments on two real datasets demonstrate that MeMix achieves superior performance compared with state-of-the-art report generation models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation cites When Radiology Report Generation Meets Knowledge Graph
  • PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation cites Generating Radiology Reports via Memory-driven Transformer
  • PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation cites Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation
  • Bootstrapping Large Language Models for Radiology Report Generation cites Generating Radiology Reports via Memory-driven Transformer
  • Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation cites Generating Radiology Reports via Memory-driven Transformer
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Generating Radiology Reports via Memory-driven TransformerZhihong Chen, Yan Song, Tsung-Hui Chang, Xiang WanEMNLP 2020 · 552 citations
  2. When Radiology Report Generation Meets Knowledge GraphYixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu et al.AAAI 2020 · 391 citations
  3. PromptMRG: Diagnosis-Driven Prompts for Medical Report GenerationHaibo Jin, Haoxuan Che, Yi Lin, Hao ChenAAAI 2024 · 168 citations
  4. Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports GenerationBin Yan, Mingtao PeiAAAI 2022 · 138 citations
  5. Bootstrapping Large Language Models for Radiology Report GenerationChang Liu, Yuanhe Tian, Weidong Chen, Yan Song et al.AAAI 2024 · 84 citations

Related papers

  1. MMTN: Multi-Modal Memory Transformer Network for Image-Report Consistent Medical Report GenerationYiming Cao, Lizhen Cui, Lei Zhang, Fuqiang Yu et al.AAAI 2023 · 56 citations
  2. DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing ModalitiesNagur Shareef Shaik, Teja Krishna Cherukuri, Adnan Masood, Dong Hye YeAAAI 2026
  3. Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEXun Zhu, Ying Hu, Fanbin Mo, Miao Li et al.NeurIPS 2024 · 29 citations
  4. Medical Report Generation via Multimodal Spatio-Temporal FusionXin Mei, Rui Mao, Xiaoyan Cai, Libin Yang et al.ACM MM 2024 · 8 citations
  5. Auto-Encoding Knowledge Graph for Unsupervised Medical Report GenerationFenglin Liu, Chenyu You, Xian Wu, Shen Ge et al.NeurIPS 2021 · 144 citations
Dusk over the sea between two cliffs drawn in fine vertical lines