Lune

EMNLP Findings2025

Lost in Embeddings: Information Loss in Vision-Language Models

Wenyan Li, Raphael Tang, Chengzu Li, Caiqi Zhang, Ivan Vulic, Anders Søgaard

Abstract

Vision--language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model's embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40--60% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Vision-Language Models Get Lost in Attention cites Visual Instruction Tuning
  • Large Vision-Language Models Get Lost in Attention cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • What matters when building vision-language models? cites Visual Instruction Tuning
  • What matters when building vision-language models? cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • What matters when building vision-language models? cites Perceiver: General Perception with Iterative Attention
  • V-LynX: Token Interface Alignment for Video+X LLMs cites Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Large Vision-Language Models Get Lost in AttentionGongli Xi, Ye Tian, Mengyu Yang, Huahui Yi et al.ICML 2026 · 4 citations
  2. SimLBR: Learning to Detect Fake Images by Learning to Detect Real ImagesAayush Dhakal, Subash Khanal, Srikumar Sastry, Jacob Arndt et al.CVPR 2026 · 1 citation
  3. AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language ModelsYue Li, Xin Yi, Dongsheng Shi, Yongyi Cui et al.KDD 2026 · 1 citation
  4. V-LynX: Token Interface Alignment for Video+X LLMsJungin Park, Jiyoung Lee, Kwanghoon SohnICML 2026

Builds on12

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  3. Perceiver: General Perception with Iterative AttentionAndrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals et al.ICML 2021 · 1,399 citations
  4. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation LearningWeixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung et al.NeurIPS 2022 · 834 citations
  5. What matters when building vision-language models?Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor SanhNeurIPS 2024 · 401 citations

Related papers

  1. Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language ModelsJiachen Jiang, Jinxin Zhou, Bo Peng, Xia Ning et al.NeurIPS 2025 · 8 citations
  2. Bridging Vision and Language Spaces with Assignment PredictionJungin Park, Jiyoung Lee, Kwanghoon SohnICLR 2024 · 15 citations
  3. AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document UnderstandingAhmed Masry, Juan A. Rodríguez, Tianyu Zhang, Suyuchen Wang et al.NeurIPS 2025 · 7 citations
  4. LatentLens: Revealing Highly Interpretable Visual Tokens in LLMsBenno Krojer, Perampalli Shravan Nayak, Oscar Mañas, Vaibhav Adlakha et al.ICML 2026 · 6 citations
  5. Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language ModelsEnguang Wang, Qiang Wang, Yuanchen Wu, Ke Yan et al.CVPR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines