Exposing and Evaluating Hallucinations for GUI Grounding

Existing GUI benchmarks primarily focus on evaluating models' comprehensive capabilities but largely overlook hallucination phenomena in grounding tasks, which are crucial to the reliability of GUI understanding. In this work, we expose two major types of hallucinations in GUI grounding: 1) Confusion Hallucination, where distractor elements are mistakenly selected, and 2) Fabricated Hallucination, where nonexistent elements are hallucinated with plausible coordinates. To systematically investigate their origins, we introduce GUI-HalluBench, a benchmark comprising two complementary subsets: a parsing subset for assessing structural representation of GUI elements and a hallucination subset for measuring grounding robustness under challenging conditions. To further address these hallucinations, we propose two approaches for alleviating hallucination behaviors: a training-free Parsing-guided Prompt (PGP) approach and a trainingbased Hallucination-aware Fine-Tuning (HFT) approach. Experiments on state-of-the-art models confirm that deficiencies in parsing lead to both confusion and fabricated hallucinations, while our easing strategies effectively reduce these hallucinations, enhancing grounding reliability. Data available at https://github.com/aiben-ch/GUI-HalluBench.