Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
Shuliang Liu, Songbo Yang, Dong Fang, Sihang Jia, Yuqi Tang, Lingfeng Su, Ruoshui Peng, Yibo Yan, Xin Zou, Xuming Hu
Abstract
Object hallucination critically undermines the reliability of Multimodal Large Language Models, often stemming from a fundamental failure in cognitive introspection, where models blindly trust linguistic priors over specific visual evidence. Existing mitigations remain limited: contrastive decoding approaches operate superficially without rectifying internal semantic misalignments, while current latent steering methods rely on static vectors that lack instance-specific precision. We introduce Vision-Language Introspection (VLI), a training-free inference framework that simulates a metacognitive self-correction process. VLI first performs Attributive Introspection to diagnose hallucination risks via probabilistic conflict detection and localize the causal visual anchors. It then employs Interpretable Bi-Causal Steering to actively modulate the inference process, dynamically isolating visual evidence from background noise while neutralizing blind confidence through adaptive calibration. VLI achieves state-of-the-art performance on advanced models, reducing object hallucination rates by 12.67% on MMHal-Bench and improving accuracy by 5.8% on POPE.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 5c22d54e-c6f4-4c24-acff-2202822ccf1eCited by top-tier papers1
Ask how each one uses itBuilds on31
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
- Analyzing and Mitigating Object Hallucination in Large Vision-Language ModelsYiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang et al.ICLR 2024 · 316 citations
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast DecodingZhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao et al.ICML 2024 · 164 citations
- Mitigating Object Hallucination via Concentric Causal AttentionYun Xing, Yiheng Li, Ivan Laptev, Shijian LuNeurIPS 2024 · 78 citations
- Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced OptimizationXinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen et al.NeurIPS 2024 · 59 citations
Related papers
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMsLiu Yu, Zhonghao Chen, Ping Kuang, Zhikun Feng et al.AAAI 2026
- Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State InterventionJingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen et al.ACL 2025 · 24 citations
- DiVE: Decoupling Intra-layer Visual Evidence for Mitigating Hallucinations in Large Vision-Language ModelsXinwei Li, Li Lin, Hui Jiao, Li Yao et al.ACL 2026
- ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLMYujun Wang, Aniri, Jinhe Bi, Sören Pirk et al.AAAI 2026 · 27 citations
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive DecodingSicong Leng, Hang Zhang, Guanzheng Chen, Xin Li et al.CVPR 2024
