Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models
Runhe Lai, Xinhua Lu, Yanqi Wu, Jinlun Ye, Weijiang Yu, Ruixuan Wang
Abstract
Multimodal large language models (MLLMs) have achieved remarkable progress, yet the object hallucination remains a critical challenge for reliable deployment. In this paper, we present an in-depth analysis of instruction token embeddings and reveal that they implicitly encode visual information while effectively filtering erroneous information introduced by misleading visual embeddings. Building on this insight, we propose the Instruction Lens Score (InsLen), which combines a Calibrated Local Score with a Context Consistency Score that measures context consistency of the object tokens. The proposed approach serves as a plug-and-play object hallucination detector without relying on auxiliary models or additional training. Extensive experiments across multiple benchmarks and diverse MLLM architectures demonstrate that In-sLen consistently outperforms existing hallucination detection methods, highlighting its effectiveness and robustness. The code is available at https://github.com/Fraserlairh/ Instruction-Lens-Score .
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext e2ca90b2-47fa-4d02-b22c-657a62a9dcfdBuilds on17
- Objects365: A Large-Scale, High-Quality Dataset for Object DetectionShuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng et al.ICCV 2019 · 1,018 citations
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction TuningFuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang et al.ICLR 2024 · 476 citations
- Uncertainty Estimation in Autoregressive Structured PredictionAndrey Malinin, Mark J. F. GalesICLR 2021 · 439 citations
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
- Detecting and Preventing Hallucinations in Large Vision Language ModelsAnisha Gunjal, Jihan Yin, Erhan BasAAAI 2024 · 312 citations
Related papers
- PAS: Prelim Attention Score for Detecting Object Hallucinations in Large Vision-Language ModelsNhat Hoang, Minh Vu, My T. Thai, Manish BhattaraiCVPR 2026 · 1 citation
- Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object HallucinationZichuan Wang, Songlin Yang, Bo Peng, Zhenchen Tang et al.CVPR 2026 · 4 citations
- ObjecTok: Learning Holistic and Robust Object Tokens for MLLMsSihan Wang, Xiyao Liu, Lianqing Liu, Zhi HanAAAI 2026
- Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual GroundingPinxue Guo, Chongruo Wu, Xinyu Zhou, Lingyi Hong et al.AAAI 2026
- Hallucinatory Image Tokens: A Training-Free EAZY Approach to Detecting and Mitigating Object Hallucinations in LVLMsLiwei Che, Tony Qingze Liu, Jing Jia, Weiyi Qin et al.ICCV 2025 · 2 citations
