Lune

ACL Findings2025

MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression

Linjie Mu, Zhongzhen Huang, Shengqian Qin, Yakun Zhu, Shaoting Zhang, Xiaofan Zhang

Abstract

Large vision-language models (LVLMs) have shown great promise in medical applications, particularly in visual question answering (Med-VQA) and diagnosis from medical images. However, existing datasets and models often fail to consider critical aspects of medical diagnostics, such as the integration of historical records and the analysis of disease progression over time. In this paper, we introduce MMXU (Multimodal and MultiX-ray Understanding), a novel dataset for MedVQA that focuses on identifying changes in specific regions between two patient visits. Unlike previous datasets that primarily address single-image questions, MMXU enables multi-image questions, incorporating both current and historical patient data. We demonstrate the limitations of current LVLMs in identifying disease progression on MMXU-test, even those that perform well on traditional benchmarks. To address this, we propose a MedRecord-Augmented Generation (MAG) approach, incorporating both global and regional historical records. Our experiments show that integrating historical records significantly enhances diagnostic accuracy by at least 20%, bridging the gap between current LVLMs and human expert performance. Additionally, we fine-tune models with MAG on MMXU-dev, which demonstrates notable improvements. We hope this work could illuminate the avenue of advancing the use of LVLMs in medical diagnostics by emphasizing the importance of historical context in interpreting medical images. Our dataset is released at github * .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • Medic-AD: Towards Medical Vision-Language Model's Clinical Intelligence cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • Medic-AD: Towards Medical Vision-Language Model's Clinical Intelligence cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • CogVLM: Visual Expert for Pretrained Language Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • CogVLM: Visual Expert for Pretrained Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • CogVLM: Visual Expert for Pretrained Language Models cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • Flamingo: a Visual Language Model for Few-Shot Learning cites ZeRO: memory optimizations toward training trillion parameter models
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Medic-AD: Towards Medical Vision-Language Model's Clinical IntelligenceWoohyeon Park, Jaeik Kim, Sunghwan Steve Cho, Pa Hong et al.CVPR 2026

Builds on9

  1. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  2. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  3. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  4. CogVLM: Visual Expert for Pretrained Language ModelsWeihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong et al.NeurIPS 2024 · 858 citations
  5. ZeRO: memory optimizations toward training trillion parameter modelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong HeSC 2020 · 852 citations

Related papers

  1. GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-Ray DiagnosisBo Liu, Ke Zou, Li-Ming Zhan, Zexin Lu et al.ICCV 2025 · 10 citations
  2. A Structured, Tagged, and Localized Visual Question Answering Dataset with Full Sentence Answers and Scene Graphs for Chest X-ray ImagesPhilip Müller, Friederike Jungmann, Georgios Kaissis, Daniel RueckertICLR 2026
  3. OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMYutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao et al.CVPR 2024
  4. MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelsMohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, Mahdi SoltanolkotabiICLR 2025
  5. GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AITianbin Li, Yanzhou Su, Wei Li, Bin Fu et al.AAAI 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines