EMNLP Findings2025
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning
Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, Jinsong Su
Abstract
Universal multimodal embedding models play a critical role in tasks such as interleaved imagetext retrieval, multimodal RAG, and multimodal clustering. However, our empirical results indicate that existing LMM-based embedding models trained with the standard InfoNCE loss exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. To deal with this issue, we propose a simple yet effective framework that dynamically improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. Within this framework, we train a series of models, named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks and 36 datasets. Experimental results show that LLaVE establishes stronger baselines that achieve state-of-the-art (SOTA) performance while demonstrating strong scalability and efficiency. Specifically, LLaVE-2B surpasses the previous SOTA 7B models, while LLaVE-7B achieves a further performance improvement of 6.2 points. Although LLaVE is trained on image-text data, it can generalize to text-video retrieval tasks in a zero-shot manner and achieve strong performance, demonstrating its remarkable potential for transfer to other embedding tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
21 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction cites Learning Transferable Visual Models From Natural Language Supervision
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction cites Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
- UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning cites Visual Instruction Tuning
- UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning cites Learning Transferable Visual Models From Natural Language Supervision
- Think Then Embed: Generative Context Improves Multimodal Embedding cites Learning Transferable Visual Models From Natural Language Supervision
- Think Then Embed: Generative Context Improves Multimodal Embedding cites Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
- Learning Transferable Visual Models From Natural Language Supervision cites A Simple Framework for Contrastive Learning of Visual Representations
- Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining cites A Simple Framework for Contrastive Learning of Visual Representations
- Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining cites Learning Transferable Visual Models From Natural Language Supervision
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings cites Visual Instruction Tuning
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings cites Learning Transferable Visual Models From Natural Language Supervision
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings cites Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
Cited by top-tier papers21
- Think Then Embed: Generative Context Improves Multimodal EmbeddingXuanming Cui, Jianpeng Cheng, Hong-You Chen, Satya Narayan Shukla et al.ICLR 2026 · 41 citations
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late InteractionZilin Xiao, Qi Ma, Mengting Gu, Chun-cheng Jason Chen et al.ICLR 2026 · 40 citations
- UME-R1: Exploring Reasoning-Driven Generative Multimodal EmbeddingsZhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou et al.ICLR 2026 · 38 citations
- Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch MiningRaghuveer Thirukovalluru, Rui Meng, Ye Liu, Karthikeyan K et al.NeurIPS 2025 · 30 citations
- UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding LearningTiancheng Gu, Kaicheng Yang, Kaichen Zhang, Xiang An et al.AAAI 2026 · 24 citations
Builds on27
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
Related papers
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding TasksZiyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz et al.ICLR 2025
- Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMsTiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang et al.ACM MM 2025 · 6 citations
- U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMsXiaojie Li, Chu Li, Shi-Zhe Chen, Xi ChenICLR 2026 · 10 citations
- Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMSSheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin et al.ICLR 2025
- ReMatch: Boosting Representation through Matching for Multimodal RetrievalQianying Liu, Xiao Liang, Zhiqiang Zhang, Yibo Chen et al.CVPR 2026 · 8 citations
