Mitigating Action-Relation Hallucinations in LVLMs via Relation-aware Visual Enhancement
Zhenxin Qin, Qiang Li, Qingzhuo Wang, Ruiyang Qin, Zhihua Wei, Wen Shen
Abstract
Large Vision-Language Models (LVLMs) have achieved remarkable performance on diverse vision-language tasks. However, LVLMs still suffer from hallucinations, generating text that contradicts the visual input. Existing research has primarily focused on mitigating object hallucinations, but often overlooks more complex relation hallucinations, particularly action relations involving interactions between objects. In this study, we empirically observe that the primary cause of action-relation hallucinations in LVLMs is the insufficient attention allocated to visual information. Thus, we propose a framework to locate action-relevant image regions and enhance the LVLM's attention to those regions. Specifically, we define the Action-Relation Sensitivity (ARS) score to identify attention heads that are most sensitive to action-relation changes, thereby localizing action-relevant image regions that contain key visual cues. Then, we propose the Relation-aware Visual Enhancement (RVE) method to enhance the LVLM's attention to these action-relevant image regions. Extensive experiments demonstrate that, compared to existing baselines, our method achieves superior performance in mitigating action-relation hallucinations with negligible additional inference cost. Furthermore, it effectively generalizes to spatial-relation hallucinations and object hallucinations. The code is available at https://github.com/LandHqzx/ARS-RVE .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding cites Evaluating Object Hallucination in Large Vision-Language Models
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding cites Detecting and Preventing Hallucinations in Large Vision Language Models
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding cites Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
- Detecting and Preventing Hallucinations in Large Vision Language Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Detecting and Preventing Hallucinations in Large Vision Language Models cites Evaluating Object Hallucination in Large Vision-Language Models
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models cites Evaluating Object Hallucination in Large Vision-Language Models
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models cites Detecting and Preventing Hallucinations in Large Vision Language Models
Builds on21
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
- Analyzing and Mitigating Object Hallucination in Large Vision-Language ModelsYiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang et al.ICLR 2024 · 316 citations
- Detecting and Preventing Hallucinations in Large Vision Language ModelsAnisha Gunjal, Jihan Yin, Erhan BasAAAI 2024 · 312 citations
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast DecodingZhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao et al.ICML 2024 · 164 citations
Related papers
- VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive SteeringZihu Wang, Boxun Xu, Yuxuan Xia, Peng LiCVPR Findings 2026
- Cracking the Code of Hallucination in LVLMs with Vision-aware Head DivergenceJinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang et al.ACL 2025
- Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object HallucinationZichuan Wang, Songlin Yang, Bo Peng, Zhenchen Tang et al.CVPR 2026 · 4 citations
- DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language ModelsYudong Zhang, Ruobing Xie, Xingwu Sun, Yiqing Huang et al.ACM MM 2025 · 1 citation
- Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate HallucinationYangneng Chen, Junlin Li, Weijun Yao, Xilai Ma et al.ACL 2026 · 1 citation
