Lune

EMNLP Findings2025

Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow

Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng

Abstract

Due to the unidirectional masking mechanism, Decoder-Only models propagate information from left to right. LVLMs (Large Vision-Language Models) follow the same architecture, with visual information gradually integrated into semantic representations during forward propagation. Through systematic analysis, we observe that the majority of the visual information is absorbed into the semantic representations. However, the model's attention distribution does not exhibit sufficient emphasis on semantic representations. This misalignment between the attention distribution and the actual information flow undermines the model's visual understanding ability and contributes to hallucinations. To address this issue, we enhance the model's visual understanding by leveraging the core information embedded in semantic representations. Specifically, we identify attention heads that focus on core semantic representations based on their attention distributions. Then, through a twostage optimization paradigm, we propagate the advantages of these attention heads across the entire model, aligning the attention distribution with the actual information flow. We evaluate our method on three image captioning benchmarks using five different LVLMs, demonstrating its effectiveness in significantly reducing hallucinations. Further experiments reveal a trade-off between reduced hallucinations and richer details. Notably, our method allows for manual adjustment of the model's conservativeness, enabling flexible control to meet diverse real-world requirements. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding cites Evaluating Object Hallucination in Large Vision-Language Models
  • Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding cites Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  • Detecting and Preventing Hallucinations in Large Vision Language Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • Detecting and Preventing Hallucinations in Large Vision Language Models cites Evaluating Object Hallucination in Large Vision-Language Models
  • Analyzing and Mitigating Object Hallucination in Large Vision-Language Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • Analyzing and Mitigating Object Hallucination in Large Vision-Language Models cites Evaluating Object Hallucination in Large Vision-Language Models
  • Analyzing and Mitigating Object Hallucination in Large Vision-Language Models cites Detecting and Preventing Hallucinations in Large Vision Language Models
  • Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention cites Evaluating Object Hallucination in Large Vision-Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided AttentionJianfei Zhao, Feng Zhang, Xin Sun, Chong Feng et al.CVPR 2026 · 6 citations
  2. Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM DecodingLiu Yu, Can Chen, PING KUANG, Zhikun Feng et al.ICML 2026

Builds on16

  1. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  2. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  3. Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
  4. Analyzing and Mitigating Object Hallucination in Large Vision-Language ModelsYiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang et al.ICLR 2024 · 316 citations
  5. Detecting and Preventing Hallucinations in Large Vision Language ModelsAnisha Gunjal, Jihan Yin, Erhan BasAAAI 2024 · 312 citations

Related papers

  1. Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention LensZhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo et al.CVPR 2025
  2. VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive SteeringZihu Wang, Boxun Xu, Yuxuan Xia, Peng LiCVPR Findings 2026
  3. Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and InterventionTianyun Yang, Ziniu Li, Juan Cao, Chang XuICLR 2025
  4. Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language ModelsXinlong Chen, Yuanxing Zhang, Qiang Liu, Junfei Wu et al.ACL Findings 2025 · 2 citations
  5. Cracking the Code of Hallucination in LVLMs with Vision-aware Head DivergenceJinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang et al.ACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines