Lune

EMNLP Findings2023

Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection

Kishan Maharaj, Ashita Saxena, Raja Kumar, Abhijit Mishra, Pushpak Bhattacharyya

Abstract

Detecting hallucinations in natural language processing (NLP) is a critical undertaking that demands a deep understanding of both the semantic and pragmatic aspects of languages. Cognitive approaches that leverage users' behavioural signals, such as gaze, have demonstrated effectiveness in addressing NLP tasks with similar linguistic complexities. However, their potential in the context of hallucination detection remains largely unexplored. In this paper, we propose a novel cognitive approach for hallucination detection that leverages gaze signals from humans. We first collect and introduce an eye tracking corpus (IITB-HGC: IITB-Hallucination Gaze corpus) consisting of 500 instances, annotated by five annotators for hallucination detection. Our analysis reveals that humans selectively attend to relevant parts of the text based on distributional similarity, similar to the attention bias phenomenon in psychology. We identify two attention strategies employed by humans: global attention, which focuses on the most informative sentence, and local attention, which focuses on important words within a sentence. Leveraging these insights, we propose a novel cognitive framework for hallucination detection that incorporates these attention biases. Experimental evaluations on the FactCC dataset demonstrate the efficacy of our approach, obtaining a balanced accuracy of 87.1%. Our study highlights the potential of gaze-based approaches in addressing the task of hallucination detection and sheds light on the cognitive processes employed by humans in identifying inconsistencies.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models cites Retrieval Augmentation Reduces Hallucination in Conversation
  • An Audit on the Perspectives and Challenges of Hallucinations in NLP cites SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
  • An Audit on the Perspectives and Challenges of Hallucinations in NLP cites Retrieval Augmentation Reduces Hallucination in Conversation
  • ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries cites SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
  • How Language Model Hallucinations Can Snowball cites Retrieval Augmentation Reduces Hallucination in Conversation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. An Audit on the Perspectives and Challenges of Hallucinations in NLPPranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs et al.EMNLP 2024 · 8 citations
  2. ETF: An Entity Tracing Framework for Hallucination Detection in Code SummariesKishan Maharaj, Vitobha Munigala, Srikanth G. Tamilselvam, Prince Kumar et al.ACL 2025 · 4 citations
  3. Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language ModelsÁngela López-Cardona, Carlos Segura, Alexandros Karatzoglou, Sergi Abadal et al.ICLR 2025

Builds on20

  1. MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
  2. Retrieval Augmentation Reduces Hallucination in ConversationKurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela et al.EMNLP Findings 2021 · 579 citations
  3. How Language Model Hallucinations Can SnowballMuru Zhang, Ofir Press, William Merrill, Alisa Liu et al.ICML 2024 · 406 citations
  4. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language ModelsPotsawee Manakul, Adian Liusie, Mark J. F. GalesEMNLP 2023 · 331 citations
  5. Asking and Answering Questions to Evaluate the Factual Consistency of SummariesAlex Wang, Kyunghyun Cho, Mike LewisACL 2020 · 317 citations

Related papers

  1. G2LDetect: A Global-to-Local Approach for Hallucination DetectionXiaoxia Cheng, Zeqi Tan, Zhe Zheng, Weiming LuAAAI 2025 · 1 citation
  2. DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language ModelsYudong Zhang, Ruobing Xie, Xingwu Sun, Yiqing Huang et al.ACM MM 2025 · 1 citation
  3. Human Attention in Image Captioning: Dataset and AnalysisSen He, Hamed Rezazadegan Tavakoli, Ali Borji, Nicolas PugeaultICCV 2019 · 55 citations
  4. Improving Natural Language Processing Tasks with Human Gaze-Guided Neural AttentionEkta Sood, Simon Tannert, Philipp Müller, Andreas BullingNeurIPS 2020 · 91 citations
  5. LAFaCT: Attribution-based Localization and Focused Sequential Analysis of Fact-Critical Tokens for Hallucination DetectionXin Wang, Jiahao Li, Licheng Zhang, Zhendong MaoACL 2026
Dusk over the sea between two cliffs drawn in fine vertical lines