Lune

CVPR Findings2026

HAIT: Hybrid Adversarial Iterative Training for Mitigating Object Hallucination in Large Vision–Language Models

Liangjie Zhao, Liao Wenjie, Ming Feng, Xiaohui Song, Huafei Li, Haonan Lu

Abstract

Large vision-language models (LVLMs) often hallucinate objects due to language priors, including spurious cooccurrence patterns learned from pure text corpora. We introduce Hybrid Adversarial Iterative Training (HAIT), a reinforcement learning-based training framework that mitigates object hallucination by directly optimizing model parameters rather than relying on heuristics at decoding time. HAIT employs a hybrid reward that balances visual faithfulness, linguistic fluency, and output diversity, mitigating the influence of language priors by explicitly penalizing unsupported object mentions. HAIT uses intermediate models from the training process for adversarial training, resulting in a more informative training signal. Through phased iterative training, HAIT suppresses co-occurrence shortcuts and steers generation toward evidence-consistent descriptions. On two widely used benchmarks, Flickr30k and HA-DPO, HAIT reduces object hallucination while improving text quality. These findings highlight HAIT as an effective approach for enhancing text generation performance with reduced object hallucination.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty DecodingFan Yuan, Chi Qin, Xiaogang Xu, Piji LiEMNLP 2024 · 1 citation
  2. Collaboration Wins More: Dual-Modal Collaborative Attention Reinforcement for Mitigating Large Vision Language Models HallucinationJiye Xie, Yifei Gao, Liangliang You, Xiang Xu et al.ACM MM 2025 · 2 citations
  3. Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination MitigationXingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang et al.ICLR 2026 · 9 citations
  4. Cracking the Code of Hallucination in LVLMs with Vision-aware Head DivergenceJinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang et al.ACL 2025
  5. AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLMLian Zhong, Ziqiang He, Jibin Zheng, Jin Li et al.CVPR 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines