ACL Findings2025
Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
Che Liu, Zhongwei Wan, Haozhe Wang, Yinda Chen, Talha Qaiser, Chen Jin, Nikolay Burlutskiy, Fariba Yousefi, Rossella Arcucci
Abstract
Vision-Language Pre-training (MedVLP) has made significant progress in enabling zero-shot tasks for medical image understanding. However, training MedVLP models typically requires large-scale datasets with paired, high-quality image-text data, which are scarce in the medical domain. Recent advancements in Large Language Models (LLMs) and diffusion models have made it possible to generate large-scale synthetic image-text pairs. This raises the question: Can MedVLP succeed using purely synthetic data? To address this, we use off-the-shelf generative models to create synthetic radiology reports and paired Chest X-ray (CXR) images, and propose an automated pipeline to build a diverse, high-quality synthetic dataset, enabling a rigorous study that isolates model and training settings, focusing entirely from the data perspective. Our results show that MedVLP models trained exclusively on synthetic data outperform those trained on real data by 3.8% in averaged AUC on zero-shot classification. Moreover, using a combination of synthetic and real data leads to a further improvement of 9.07%. Additionally, MedVLP models trained on synthetic or mixed data consistently outperform those trained on real data in zero-shot grounding, as well as in fine-tuned classification and segmentation tasks. Our analysis suggests MedVLP trained on well-designed synthetic data can outperform models trained on real datasets, which may be limited by lowquality samples and long-tailed distributions. The code will be released at https:// github.com/cheliu-computation/ MedSyn-RepLearn.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society cites High-Resolution Image Synthesis with Latent Diffusion Models
- Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning cites GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition
- How Far Can We Go With Synthetic Data for Audio-Visual Sound Source Localization? cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers1
Builds on20
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyGuohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin et al.NeurIPS 2023 · 1,975 citations
- GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image RecognitionShih-Cheng Huang, Liyue Shen, Matthew P. Lungren, Serena YeungICCV 2021 · 516 citations
- Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation LearningFuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti et al.NeurIPS 2022 · 302 citations
Related papers
- PairAug: What Can Augmented Image-Text Pairs Do for Radiology?Yutong Xie, Qi Chen, Sinuo Wang, Minh-Son To et al.CVPR 2024 · 6 citations
- Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed TomographyBowen Shi, Weiwei Cao, Ruifeng Yuan, Wanxing Chang et al.ICML 2026
- MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training for X-ray DiagnosisChaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang et al.ICCV 2023 · 205 citations
- MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual RepresentationsZiyang Zhang, Yang Yu, Yucheng Chen, Xulei Yang et al.CVPR 2025
- MEDICAL IMAGE UNDERSTANDING WITH PRETRAINED VISION LANGUAGE MODELS: A COMPREHENSIVE STUDYZiyuan Qin, Huahui Yi, Qicheng Lao, Kang LiICLR 2023 · 25 citations
