Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models
Runhe Lai, Xinhua Lu, Yanqi Wu, Jinlun Ye, Weijiang Yu, Ruixuan Wang
摘要
Multimodal large language models (MLLMs) have achieved remarkable progress, yet the object hallucination remains a critical challenge for reliable deployment. In this paper, we present an in-depth analysis of instruction token embeddings and reveal that they implicitly encode visual information while effectively filtering erroneous information introduced by misleading visual embeddings. Building on this insight, we propose the Instruction Lens Score (InsLen), which combines a Calibrated Local Score with a Context Consistency Score that measures context consistency of the object tokens. The proposed approach serves as a plug-and-play object hallucination detector without relying on auxiliary models or additional training. Extensive experiments across multiple benchmarks and diverse MLLM architectures demonstrate that In-sLen consistently outperforms existing hallucination detection methods, highlighting its effectiveness and robustness. The code is available at https://github.com/Fraserlairh/ Instruction-Lens-Score .
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了最后一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper17
- Objects365: A Large-Scale, High-Quality Dataset for Object DetectionShuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng 等ICCV 2019 · 被引用 1,018 次
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction TuningFuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang 等ICLR 2024 · 被引用 476 次
- Uncertainty Estimation in Autoregressive Structured PredictionAndrey Malinin, Mark J. F. GalesICLR 2021 · 被引用 439 次
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang 等EMNLP 2023 · 被引用 344 次
- Detecting and Preventing Hallucinations in Large Vision Language ModelsAnisha Gunjal, Jihan Yin, Erhan BasAAAI 2024 · 被引用 312 次
相关 Paper
- PAS: Prelim Attention Score for Detecting Object Hallucinations in Large Vision-Language ModelsNhat Hoang, Minh Vu, My T. Thai, Manish BhattaraiCVPR 2026 · 被引用 1 次
- Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object HallucinationZichuan Wang, Songlin Yang, Bo Peng, Zhenchen Tang 等CVPR 2026 · 被引用 4 次
- ObjecTok: Learning Holistic and Robust Object Tokens for MLLMsSihan Wang, Xiyao Liu, Lianqing Liu, Zhi HanAAAI 2026
- Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual GroundingPinxue Guo, Chongruo Wu, Xinyu Zhou, Lingyi Hong 等AAAI 2026
- Hallucinatory Image Tokens: A Training-Free EAZY Approach to Detecting and Mitigating Object Hallucinations in LVLMsLiwei Che, Tony Qingze Liu, Jing Jia, Weiyi Qin 等ICCV 2025 · 被引用 2 次
