ACL Findings2026
KnowDR-REC: Auditing Knowledge-Conditioned Visual Grounding in Referring Expression Comprehension
Guanghao Jin, Jingpei Wu, Tianpei Guo, Yiyi Niu, Weidong Zhou, Linyi Yang, Guoyang Liu
Abstract
While Multimodal Large Language Models (MLLMs) have demonstrated the capacity for multi-modal reasoning, current Referring Expression Comprehension (REC) benchmarks lag behind, predominantly relying on intra-image cues and neglecting the integration of external world knowledge, which significantly impedes the evolution of REC towards real-world applications. This limitation obscures a model’s true capability to conduct textual reasoning (entity resolution), resolve spatial location (visual grounding), and verify reference validity (hallucination rejection). To address this, we introduce KnowDR-REC, a targeted audit benchmark comprising 1,042 positive triplets derived from real-world knowledge, along with rigorously matched negative samples. Unlike traditional datasets, we implement a controllable counterfactual evaluation mechanism that subjects textual expressions to single-factor perturbations (entity, relation, or time) to test sensitivity to fine-grained factual changes. Extensive evaluation of 18 state-of-the-art MLLMs exposes a critical “binding hallucination,” revealing that current high performance is often built on fragile visual shortcuts rather than true understanding. KnowDR-REC thus serves as a pivotal diagnostic instrument, steering future research toward the genuine integration of perception and reasoning.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Ferret: Refer and Ground Anything Anywhere at Any Granularity cites Visual Instruction Tuning
- Ferret: Refer and Ground Anything Anywhere at Any Granularity cites Evaluating Object Hallucination in Large Vision-Language Models
- Evaluating Object Hallucination in Large Vision-Language Models cites Visual Instruction Tuning
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models cites Visual Instruction Tuning
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models cites Evaluating Object Hallucination in Large Vision-Language Models
Builds on19
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanNeurIPS 2023 · 1,792 citations
- Ferret: Refer and Ground Anything Anywhere at Any GranularityHaoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du et al.ICLR 2024 · 515 citations
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language ModelsDongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang et al.ICML 2024 · 149 citations
Related papers
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression TasksQihua Dong, Kuo Yang, Lin Ju, Handong Zhao et al.ICLR 2026 · 13 citations
- FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression ComprehensionJunzhuo Liu, Xuzheng Yang, Weiwei Li, Peng WangEMNLP 2024 · 2 citations
- M³-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question AnsweringJiatong Ma, Longteng Guo, Yuchen Liu, Zijia Zhao et al.ACL 2026
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language ModelsSung-Bin Kim, Oh Hyun-Bin, JungMok Lee, Arda Senocak et al.ICLR 2025
- Multimodal Fact-Level Attribution for Verifiable ReasoningDavid Wan, Han Wang, Ziyang Wang, Elias Stengel-Eskin et al.ICML 2026 · 2 citations
