Lune

EMNLP Findings2024

MATE: Meet At The Embedding - Connecting Images with Long Texts

Young Kyun Jang, Junmo Kang, Yong Jae Lee, Donghyun Kim

Abstract

While advancements in Vision Language Models (VLMs) have significantly improved the alignment of visual and textual data, these models primarily focus on aligning images with short descriptive captions. This focus limits their ability to handle complex text interactions, particularly with longer texts such as lengthy captions or documents, which have not been extensively explored yet. In this paper, we introduce Meet At The Embedding (MATE), a novel approach that combines the capabilities of VLMs with Large Language Models (LLMs) to overcome this challenge without the need for additional image-long text pairs. Specifically, we replace the text encoder of the VLM with a pretrained LLM-based encoder that excels in understanding long texts. To bridge the gap between VLM and LLM, MATE incorporates a projection module that is trained in a multi-stage manner. It starts by aligning the embeddings from the VLM text encoder with those from the LLM using extensive text pairs. This module is then employed to seamlessly align image embeddings closely with LLM embeddings. We propose two new cross-modal retrieval benchmarks to assess the task of connecting images with long texts (lengthy captions / documents). Extensive experimental results demonstrate that MATE effectively connects images with long texts, uncovering diverse semantic relationships.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • Generalized Contrastive Learning for Universal Multimodal Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
  • StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues cites LoRA: Low-Rank Adaptation of Large Language Models
  • StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues cites Learning Transferable Visual Models From Natural Language Supervision
  • VAEmo: Efficient Representation Learning for Visual-Audio Emotion With Knowledge Injection cites LoRA: Low-Rank Adaptation of Large Language Models
  • VAEmo: Efficient Representation Learning for Visual-Audio Emotion With Knowledge Injection cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites A Simple Framework for Contrastive Learning of Visual Representations
  • Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence cites Learning Transferable Visual Models From Natural Language Supervision
  • Panoptic Captioning: An Equivalence Bridge for Image and Text cites Visual Instruction Tuning
  • Panoptic Captioning: An Equivalence Bridge for Image and Text cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Generalized Contrastive Learning for Universal Multimodal RetrievalJungsoo Lee, Janghoon Cho, Hyojin Park, Durga Malladi et al.NeurIPS 2025 · 11 citations
  2. VAEmo: Efficient Representation Learning for Visual-Audio Emotion With Knowledge InjectionHao Cheng, Zhiwei Zhao, Yichao He, Zhenzhen Hu et al.ACM MM 2025 · 9 citations
  3. Distributional Vision-Language Alignment by Cauchy-Schwarz DivergenceWenzhe Yin, Zehao Xiao, Pan Zhou, Shujian Yu et al.ICLR 2026 · 9 citations
  4. Panoptic Captioning: An Equivalence Bridge for Image and TextKun-Yu Lin, Hongjun Wang, Weining Ren, Kai HanNeurIPS 2025 · 7 citations
  5. StructXLIP: Enhancing Vision-language Models with Multimodal Structural CuesZanxi Ruan, Songqun Gao, Qiuyu Kong, Yiming Wang et al.CVPR 2026 · 1 citation

Builds on19

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  3. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  4. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  5. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations

Related papers

  1. AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document UnderstandingAhmed Masry, Juan A. Rodríguez, Tianyu Zhang, Suyuchen Wang et al.NeurIPS 2025 · 7 citations
  2. Bridging Vision and Language Spaces with Assignment PredictionJungin Park, Jiyoung Lee, Kwanghoon SohnICLR 2024 · 15 citations
  3. Token Embeddings Alignment for Cross-Modal RetrievalChen-Wei Xie, Jianmin Wu, Yun Zheng, Pan Pan et al.ACM MM 2022 · 18 citations
  4. HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language EmbeddingChenxin Tao, Shiqian Su, Xizhou Zhu, Chenyu Zhang et al.CVPR 2025
  5. Generating Images with Multimodal Language ModelsJing Yu Koh, Daniel Fried, Russ SalakhutdinovNeurIPS 2023 · 403 citations
Dusk over the sea between two cliffs drawn in fine vertical lines