VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering. Recently, there has been a surge of interest in developing universal text embedding models that can generalize across tasks (e.g., MTEB). However, progress in learning universal multimodal embedding models has been relatively slow despite its importance and practicality. In this work, we aim to explore the potential of building universal multimodal embeddings capable of handling a wide range of downstream tasks. Our contributions are two fold: (1) we propose MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training datasets and 16 evaluation datasets covering both in-distribution and out-of-distribution tasks, and (2) VL M2VE C (Vision-Language Model → Vector), a contrastive training framework that converts any visionlanguage model into an embedding model via contrastive training on MMEB. Unlike previous models such as CLIP or BLIP, which encodes text or images independently without any task instruction, VL M2VE C can process any combination of images and text to generate a fixed-dimensional vector based on the given task instructions. We build a series of VL M2VE C models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split. With LoRA tuning, VL M2VE C can achieve an improvement of 10% to 20% over existing multimodal embedding models on MMEB evaluation sets. We show that VLMs are secretly strong embedding models.