Should VLMs be Pre-trained with Image Data?

Pre-trained LLMs that are further trained with image data perform well on visionlanguage tasks. While adding images during a second training phase effectively unlocks this capability, it is unclear how much of a gain or loss this two-step pipeline gives over VLMs which integrate images earlier into the training process. To investigate this, we train models spanning various datasets, scales, image-text ratios, and amount of pre-training done before introducing vision tokens. We then fine-tune these models and evaluate their downstream performance on a suite of vision-language and text-only tasks. We find that pre-training with a mixture of image and text data allows models to perform better on vision-language tasks while maintaining strong performance on text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B model, introducing visual tokens 80% of the way through pre-training results in a 2% average improvement over introducing visual tokens to a fully pre-trained model. LLM Pre-training Image-text Pre-training Fine-tuning "Human": "What could be a potential reason for this nearly empty bowl? "GPT": "A potential reason [...] A top view of the electronic board of a computer 3.4T tokens 1B tokens DCLM DataComp-DR LLaVa Amount of training Mix of data source Data sources "Imagine processing 400-billion pieces of information per second! Sound impossible? That's exactly how fast [..