Lune

ICLR2025Top-tier venue

Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Wenze Hu, Juan Lao Tebar, Zhe Gan, Peter Grasch, Meng Cao, Yinfei Yang

Abstract

Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension cites Visual Instruction Tuning
  • Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension cites Learning Transferable Visual Models From Natural Language Supervision
  • STIV: Scalable Text and Image Conditioned Video Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
  • STIV: Scalable Text and Image Conditioned Video Generation cites Scalable Diffusion Models with Transformers
  • STIV: Scalable Text and Image Conditioned Video Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • Multimodal Autoregressive Pre-training of Large Vision Encoders cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Multimodal Autoregressive Pre-training of Large Vision Encoders cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • Multimodal Autoregressive Pre-training of Large Vision Encoders cites Learning Transferable Visual Models From Natural Language Supervision
  • SmartCLIP: Modular Vision-language Alignment with Identification Guarantees cites High-Resolution Image Synthesis with Latent Diffusion Models
  • SmartCLIP: Modular Vision-language Alignment with Identification Guarantees cites Visual Instruction Tuning
  • SmartCLIP: Modular Vision-language Alignment with Identification Guarantees cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • SmartCLIP: Modular Vision-language Alignment with Identification Guarantees cites Learning Transferable Visual Models From Natural Language Supervision
  • CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning cites Visual Instruction Tuning
  • CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning cites Learning Transferable Visual Models From Natural Language Supervision
  • Scalable Diffusion Models with Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Scalable Diffusion Models with Transformers cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image CaptioningZhijiang Tang, Linhua Wang, Jiaxin Qi, Weihao Jiang et al.CVPR 2026 · 7 citations
  2. Scaling Inference-Time Search with Vision Value Model for Improved Visual ComprehensionXiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu et al.ICCV 2025 · 1 citation
  3. STIV: Scalable Text and Image Conditioned Video GenerationZongyu Lin, Wei Liu, Chen Chen, Jiasen Lu et al.ICCV 2025 · 1 citation
  4. Multimodal Autoregressive Pre-training of Large Vision EncodersEnrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter et al.CVPR 2025
  5. SmartCLIP: Modular Vision-language Alignment with Identification GuaranteesShaoan Xie, Lingjing Kong, Yujia Zheng, Yu Yao et al.CVPR 2025

Builds on28

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  4. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
  5. Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations

Related papers

  1. Role-SynthCLIP: A Role-Play Driven Diverse Synthetic Data ApproachYuanxiang Huangfu, Chaochao wang, weilei wangCVPR 2026
  2. BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation ModelsZiheng Zhang, Xinyue Ma, Arpita Chowdhury, Elizabeth G Campolongo et al.ICLR 2026 · 3 citations
  3. CapsFusion: Rethinking Image-Text Data at ScaleQiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui et al.CVPR 2024 · 17 citations
  4. Exploring the Potential of Dense Information in Multimodal AlignmentZhiyuan Fan, Zhihong Chen, Benyou WangACL Findings 2024
  5. CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal LearningQingqing Cao, Mahyar Najibi, Sachin MehtaICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines