Lune

ACL Findings2026

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

Maharshi Gor, Yoo Yeon Sung, Yu Hou, Eve Fleisig, Zhu Irene Ying, Tianyi Zhou, Jordan Lee Boyd-Graber

Abstract

AI systems are fallible, and humans can make mistakes in deciding whether to trust AI over their own judgment. Thus, improving human-AI collaboration requires that we understand when, why, and how humans decide to rely on AI. We study two reliance decisions: delegating a task to AI without seeing its output (whether AI is used) and evaluating AI suggestions to decide whether to adopt them (how AI output shapes final decisions). Both matter for effective collaboration, yet prior work lacks naturalistic experiments capturing both patterns for the same users. We address this gap by studying collaborative human-AI teams competing in a question-answering game in which humans can choose when and how to work with AI agents to win. Our 24 matches pair 23 expert humans with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. While human-AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.7% of opportunities missed) and over-relying when AI misleads them (1.5%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, while confirmation bias drives higher under-reliance (60.7%) when an AI suggestion agrees with humans' initial incorrect answer.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance cites Manipulating and Measuring Model Interpretability
  • Expanding Explainability: Towards Social Transparency in AI systems cites Manipulating and Measuring Model Interpretability
  • To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making cites Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
  • Explanations Can Reduce Overreliance on AI Systems During Decision-Making cites Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
  • Explanations Can Reduce Overreliance on AI Systems During Decision-Making cites Expanding Explainability: Towards Social Transparency in AI systems
  • Explanations Can Reduce Overreliance on AI Systems During Decision-Making cites To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
  • Explanations Can Reduce Overreliance on AI Systems During Decision-Making cites Manipulating and Measuring Model Interpretability
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingZana Buçinca, Maja Barbara Malaya, Krzysztof Z. GajosCSCW 2021 · 962 citations
  2. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team PerformanceGagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok et al.CHI 2021 · 713 citations
  3. Manipulating and Measuring Model InterpretabilityForough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan et al.CHI 2021 · 663 citations
  4. Expanding Explainability: Towards Social Transparency in AI systemsUpol Ehsan, Q. Vera Liao, Michael J. Muller, Mark O. Riedl et al.CHI 2021 · 505 citations
  5. Explanations Can Reduce Overreliance on AI Systems During Decision-MakingHelena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg et al.CSCW 2023 · 362 citations

Related papers

  1. Understanding Choice Independence and Error Types in Human-AI CollaborationAlexander Erlei, Abhinav Sharma, Ujwal GadirajuCHI 2024 · 25 citations
  2. Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-makingCharvi Rastogi, Yunfeng Zhang, Dennis Wei, Kush R. Varshney et al.CSCW 2022 · 184 citations
  3. Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision MakingZhuoran Lu, Dakuo Wang, Ming YinCSCW 2024 · 36 citations
  4. Uncalibrated Models Can Improve Human-AI CollaborationKailas Vodrahalli, Tobias Gerstenberg, James Y. ZouNeurIPS 2022 · 47 citations
  5. As Confidence Aligns: Understanding the Effect of AI Confidence on Human Self-confidence in Human-AI Decision MakingJingshu Li, Yitian Yang, Q. Vera Liao, Junti Zhang et al.CHI 2025 · 56 citations
Dusk over the sea between two cliffs drawn in fine vertical lines