Role-SynthCLIP: A Role-Play Driven Diverse Synthetic Data Approach
Yuanxiang Huangfu, Chaochao wang, weilei wang
摘要
The effectiveness of Contrastive Language-Image Pretraining (CLIP) models critically depends on the semantic diversity and quality of their training data. However, while existing synthetic data generation methods primarily focus on increasing data volume, such emphasis often leads to limited semantic diversity and redundant or shallow captions. To address this limitation, we propose Role-SynthCLIP, a novel data synthesis framework that leverages multi-perspective role-playing prompts (e.g., a compositional analyst, an interpreter of image context) to guide Multimodal Large Language Models (MLLMs) in generating semantically diverse captions from distinct viewpoints. This mechanism enhances the semantic diversity and finegrained image-text alignment of synthetic pairs, thereby improving caption expressiveness and accuracy while keeping the total number of image-text pairs unchanged. Experimental results demonstrate the effectiveness and efficiency of our method. A CLIP-B/16 model trained on only 1 million Role-SynthCLIP pairs achieves a Recall@1 of 64.1% on the MS COCO validation set, surpassing the best existing synthetic data baseline (trained on 5M pairs) by 2.8 percentage points. The code and trained models are released at https://github.com/huangfu170/Role-SynthCLIP.
Question: What kinds of roles are usually used when generating precise i mage descriptions? Ideas can be enriched by introducing different character s' perspectives.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper10
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh 等ICML 2021 · 被引用 47,906 次
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 被引用 2,932 次
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang 等EMNLP 2023 · 被引用 344 次
- Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPThao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh 等NeurIPS 2022 · 被引用 131 次
- MagicLens: Self-Supervised Image Retrieval with Open-Ended InstructionsKai Zhang, Yi Luan, Hexiang Hu, Kenton Lee 等ICML 2024 · 被引用 112 次
相关 Paper
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data UtilizationYu Zhang, Qi Zhang, Zixuan Gong, Yiwei Shi 等ICML 2024 · 被引用 9 次
- RWKV-CLIP: A Robust Vision-Language Representation LearnerTiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng 等EMNLP 2024 · 被引用 11 次
- Multimodal Hypothetical Summary for Retrieval-based Multi-image Question AnsweringPeize Li, Qingyi Si, Peng Fu, Zheng Lin 等AAAI 2025 · 被引用 1 次
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long CaptionsZiteng Wang, Siqi Yang, Limeng Qiao, Lin MaNeurIPS 2025 · 被引用 5 次
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language NegativesMaitreya Patel, Abhiram Kusumba, Sheng Cheng, Changhoon Kim 等NeurIPS 2024 · 被引用 73 次
