When AI Gives Advice: Evaluating AI and Human Responses to Online Advice-Seeking for Well-Being
Harsh Kumar, Jasmine Chahal, Yinuo Zhao, Zeling Zhang, Annika Z. Wei, Louis Tay, Ashton Anderson
摘要
Seeking advice is a core human behavior that the internet has reinvented twice: first through forums and Q&A communities that crowdsource public guidance, and now through large language models (LLMs). Yet the quality of this LLM advice for everyday well-being scenarios remains unclear. How does it compare, not only against human comments, but against the wisdom of the online crowd? We ran two studies (N=210) in which experts compared top-voted Reddit advice with LLM-generated advice. LLMs ranked significantly higher overall and on effectiveness, warmth, and willingness to seek advice again. GPT-4o beat GPT-5 on all metrics except sycophancy, suggesting that benchmark gains need not improve advice-giving. In Study-2, we examined how human and algorithmic advice could be combined, and found that human advice can be unobtrusively polished to compete with AI-generated comments. We conclude with design implications for advice-giving agents and ecosystems blending AI, crowd input, and expert oversight.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper17
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida 等NeurIPS 2022 · 被引用 24,707 次
- Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris 等UIST 2023 · 被引用 1,882 次
- Towards Understanding Sycophancy in Language ModelsMrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud 等ICLR 2024 · 被引用 762 次
- Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future DirectionsMagdalena Wischnewski, Nicole C. Krämer, Emmanuel MüllerCHI 2023 · 被引用 135 次
- "Are You Really Sure?" Understanding the Effects of Human Self-Confidence Calibration in AI-Assisted Decision MakingShuai Ma, Xinru Wang, Ying Lei, Chuhan Shi 等CHI 2024 · 被引用 54 次
相关 Paper
- Objection Overruled! Lay People can Distinguish Large Language Models from Lawyers, but still Favour Advice from an LLMEike Schneiders, Tina Seabrooke, Joshua Krook, Richard Hyde 等CHI 2025 · 被引用 15 次
- Generating Automatic Feedback on UI Mockups with Large Language ModelsPeitong Duan, Jeremy Warner, Yang Li, Bjoern HartmannCHI 2024 · 被引用 81 次
- On the Planning Abilities of Large Language Models - A Critical InvestigationKarthik Valmeekam, Matthew Marquez, Sarath Sreedharan, Subbarao KambhampatiNeurIPS 2023 · 被引用 509 次
- How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?Ryan Liu, Theodore R. Sumers, Ishita Dasgupta, Thomas L. GriffithsICML 2024 · 被引用 33 次
- "It's the only thing I can trust": Envisioning Large Language Model Use by Autistic Workers for Communication AssistanceJiWoong Jang, Sanika Moharana, Patrick Carrington, Andrew BegelCHI 2024 · 被引用 58 次
