Lune

CVPR2026Top-tier venue

Mechanisms of Object Localization in Vision-Language Models

Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig

Abstract

Visually-grounded language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including token ablations, attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object, while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Vision Transformers Need Registers cites Learning Transferable Visual Models From Natural Language Supervision
  • Grounding Multimodal Large Language Models to the World cites Visual Instruction Tuning
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on21

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  4. Grounding Multimodal Large Language Models to the WorldZhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao et al.ICLR 2024 · 1,170 citations
  5. Vision Transformers Need RegistersTimothée Darcet, Maxime Oquab, Julien Mairal, Piotr BojanowskiICLR 2024 · 769 citations

Related papers

  1. Towards Interpreting Visual Information Processing in Vision-Language ModelsClement Neo, Luke Ong, Philip Torr, Mor Geva et al.ICLR 2025
  2. Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual GroundingSeil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae HwangCVPR 2025
  3. Interpretability Transfer from Language to Vision via Sparse AutoencodersAlexey Kravets, Da Li, Chuan Li, Da Chen et al.ICML 2026
  4. D²Pruner: Debiased Importance and Structural Diversity for MLLM Token PruningEvelyn Zhang, Fufu Yu, Aoqi Wu, Zichen Wen et al.AAAI 2026 · 1 citation
  5. Enhancing Multimodal Large Language Models Complex Reason via Similarity ComputationXiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui et al.AAAI 2025 · 36 citations
Dusk over the sea between two cliffs drawn in fine vertical lines