Characterizing User-Reported Risks across LLM Chatbots
Lingyao Li, Renkai Ma, Zhaoqian Xue, Junjie Xiong
Abstract
As Large Language Models (LLMs) become increasingly integral to daily life, users are engaging with multiple LLM chatbots for various needs; however, prior research on LLM risks often remains lab-based or focuses on single LLMs like ChatGPT or singular risks like privacy. To gain a multi-risk, cross-chatbot understanding of user experiences, we analyze Reddit discussions around seven major LLM chatbots using the NIST AI Risk Management Framework. We find that user-reported risks are unevenly distributed and chatbot-specific: ChatGPT is associated with safety and fairness concerns, Gemini with privacy, and Claude with security and resilience. Less frequent risks, such as explainability and privacy, appear as user trade-offs, whereas prevalent risks like fairness are experienced as direct harms. Our findings underscore the need to operationalize chatbot-specific risk mitigation, moving beyond system-centered risk mitigation to human-centered interventions that align with users’ lived experiences.
Ask about this paper
Ask your agent about it.
Lune has read the top-tier papers around this one, so every answer names the papers it rests on.
Your agent calls
Lunesearch_papers
Free to start. No credit card required.
Terminal
Install the CLIlune papers get 9bc75253-2b74-4997-9a51-3c3a4d2479faRelated papers
- FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMsZhiting Fan, Ruizhe Chen, Tianxiang Hu, Zuozhu LiuICLR 2025
- Exploring User Security and Privacy Attitudes and Concerns Toward the Use of General-Purpose LLM Chatbots for Mental HealthJabari Kwesi, Jiaxun Cao, Riya Manchanda, Pardis Emami NaeiniUSENIX Security 2025
- Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI PlatformsMutahar Ali, Arjun Arunasalam, Habiba FarrukhS&P 2025
- Decision-Making Behavior Evaluation Framework for LLMs under Uncertain ContextJingru Jia, Zehua Yuan, Junhao Pan, Paul McNamara et al.NeurIPS 2024 · 71 citations
- "It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational AgentsZhiping Zhang, Michelle Jia, Hao-Ping (Hank) Lee, Bingsheng Yao et al.CHI 2024 · 92 citations
