ACL Findings2026
AHEAD: Attention Head Energy-Aware Dynamics for Hallucination Mitigation in MLLMs
Jiale Chang, Ying Li, Siliang Tang, Yueting Zhuang
Abstract
Multimodal large language models excel at vision-language tasks but remain prone to hallucinations that undermine their reliability. Existing approaches predominantly treat hallucinations as classification errors, overlooking the heterogeneous behaviors of attention heads and their dynamic influences during inference. We revisit MLLM reasoning from an energy perspective and identify that hallucinations stem from imbalances between visual potential and language prior potential: when visual information is ambiguous or language priors dominate, attention heads tend to be driven by linguistic statistical patterns, generating content inconsistent with visual evidence. We propose AHEAD, a framework that quantifies the energetic properties of each attention head during object generation through two potential networks-the Visual Grounding Potential Network and the Language Prior Potential Network-and dynamically adjusts their contributions at inference time. Specifically, we amplify attention heads with strong visual grounding capacity while suppressing those overly reliant on language priors. Experiments across multiple benchmarks demonstrate that AHEAD significantly reduces hallucination rates without fine-tuning the base MLLM while maintaining generation quality.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models cites Contrastive Decoding: Open-ended Text Generation as Optimization
- Aligning Large Multimodal Models with Factually Augmented RLHF cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Aligning Large Multimodal Models with Factually Augmented RLHF cites Evaluating Object Hallucination in Large Vision-Language Models
Builds on21
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language ModelsYung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim et al.ICLR 2024 · 354 citations
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
- Aligning Large Multimodal Models with Factually Augmented RLHFZhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu et al.ACL Findings 2024 · 142 citations
- Contrastive Decoding: Open-ended Text Generation as OptimizationXiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang et al.ACL 2023 · 78 citations
Related papers
- Cracking the Code of Hallucination in LVLMs with Vision-aware Head DivergenceJinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang et al.ACL 2025
- PostAlign: Multimodal Grounding as a Corrective Lens for MLLMsYixuan Wu, Yang Zhang, Jian Wu, Philip Torr et al.ICLR 2026 · 5 citations
- DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language ModelsYudong Zhang, Ruobing Xie, Xingwu Sun, Yiqing Huang et al.ACM MM 2025 · 1 citation
- SHARP: Steering Hallucination in LVLMs via Representation EngineeringJunfei Wu, Yue Ding, Guofan Liu, Tianze Xia et al.EMNLP 2025
- MLLM can see? Dynamic Correction Decoding for Hallucination MitigationChenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian et al.ICLR 2025
