ACL Findings2026
System-Mediated Attention Imbalances Make Vision-Language Models Say Yes
Tsan Tsai Chan, Varsha Suresh, Anisha Saha, Michael Hahn, Vera Demberg
Abstract
Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text. However, existing mitigation strategies tend towards an image-centric interpretation of these imbalances, often prioritising increased image attention while giving less consideration to the roles of the other modalities. In this study, we evaluate a more holistic, system-mediated account, which attributes these imbalances to functionally redundant system weights that reduce attention to image and textual inputs. We show that this framework offers a useful empirical perspective on the yes-bias, a common form of hallucination in which VLMs indiscriminately respond 'yes'. Causally redistributing attention from the system modality to image and textual inputs substantially suppresses this bias, often outperforming existing approaches. We further present evidence suggesting that system-mediated attention imbalances contribute to the yes-bias by encouraging a default reliance on coarse input representations, which are effective for some tasks but ill-suited to others. Taken together, these findings firmly establish system attention as a key factor in VLM hallucination and highlight its potential as a lever for mitigation. The code for our main interventions can be found at:
https://github.com/anisha0325/ vlm-hallucination-yes-bias/tree/main.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Evaluating Object Hallucination in Large Vision-Language Models cites Visual Instruction Tuning
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI cites Visual Instruction Tuning
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI cites Evaluating Object Hallucination in Large Vision-Language Models
Builds on27
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIXiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang et al.CVPR 2024 · 213 citations
- Winoground: Probing Vision and Language Models for Visio-Linguistic CompositionalityTristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh et al.CVPR 2022 · 179 citations
Related papers
- Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and InterventionTianyun Yang, Ziniu Li, Juan Cao, Chang XuICLR 2025
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMsLiu Yu, Zhonghao Chen, Ping Kuang, Zhikun Feng et al.AAAI 2026
- Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination MitigationLexiang Tang, Xianwei Zhuang, Bang Yang, Zhiyuan Hu et al.AAAI 2026 · 8 citations
- Mitigating Object Hallucinations in LVLMs via Attention Imbalance RectificationHan Sun, Qin Li, Peixin Wang, Min ZhangCVPR Findings 2026
- Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise PerturbationZhenghua Wang, Yixin Wu, Feiran Zhang, Qi Qian et al.ACL Findings 2026
