EMNLP Findings2024
On Evaluating Explanation Utility for Human-AI Decision Making in NLP
Fateme Hashemi Chaleshtori, Atreya Ghosal, Alexander Gill, Purbid Bambroo, Ana Marasovic
Abstract
Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations help people in situations they are introduced for. More humancentered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies.
To aid with this, we first review existing metrics suitable for application-grounded evaluation. We then establish criteria to select appropriate datasets, and using them, we find that only 4 out of over 50 datasets available for explainability research in NLP meet them. We then demonstrate the importance of reassessing the state of the art to form and study human-AI teams: teaming people with models for certain tasks might only now start to make sense, and for others, it remains unsound. Finally, we present the exemplar studies of human-AI decision-making for one of the identified tasks -verifying the correctness of a legal claim given a contract. Our results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance. We argue for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
- Understanding the Effects of AI-Assisted Critical Thinking on Human-AI Decision MakingHarry Yizhou Tian, Hasan Amin, Ming YinCHI 2026 · 3 citations
- Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model ExplanationsKeyu He, Tejas Srinivasan, Brihi Joshi, Xiang Ren et al.ACL 2026
Builds on14
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- Explanations Can Reduce Overreliance on AI Systems During Decision-MakingHelena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg et al.CSCW 2023 · 362 citations
- ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningWeihao Yu, Zihang Jiang, Yanfei Dong, Jiashi FengICLR 2020 · 325 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
- Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their NeedsHarini Suresh, Steven R. Gomez, Kevin K. Nam, Arvind SatyanarayanCHI 2021 · 115 citations
Related papers
- What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability MethodsJulien Colin, Thomas Fel, Rémi Cadène, Thomas SerreNeurIPS 2022 · 147 citations
- On the Interaction of Belief Bias and ExplanationsAna Valeria Gonzalez, Anna Rogers, Anders SøgaardACL Findings 2021 · 10 citations
- The effectiveness of feature attribution methods and its correlation with automatic evaluation scoresGiang Nguyen, Daeyoung Kim, Anh NguyenNeurIPS 2021 · 128 citations
- Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language ExplanationsBingsheng Yao, Prithviraj Sen, Lucian Popa, James A. Hendler et al.ACL 2023 · 4 citations
- Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric AssessmentsMarharyta Domnich, Julius Välja, Rasmus Moorits Veski, Giacomo Magnifico et al.AAAI 2025 · 11 citations
