Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners
Qiaosi Wang, Jini Kim, Avanita Sharma, Alicia (Hyun Jin) Lee, Jodi Forlizzi, Hong Shen
Abstract
Theory of Mind (ToM) -- the ability to infer what others are thinking (e.g., intentions) from observable cues -- is traditionally considered fundamental to human social interactions. This has sparked growing efforts in building and benchmarking AI's ToM capability, yet little is known about how such capability could translate into the design and experience of everyday user-facing AI products and services. We conducted 13 co-design sessions with 26 U.S.-based AI practitioners to envision, reflect, and distill design recommendations for ToM-enabled everyday AI products and services that are both future-looking and grounded in the realities of AI design and development practices. Analysis revealed three interrelated design recommendations: ToM-enabled AI should 1) be situated in the social context that shape users' mental states, 2) be responsive to the dynamic nature of mental states, and 3) be attuned to subjective individual differences. We surface design tensions within each recommendation that reveal a broader gap between practitioners' envisioned futures of ToM-enabled AI and the realities of current AI design and development practices. These findings point toward the need to move beyond static, inference-driven approach to ToM and toward designing ToM as a pervasive capability that supports continuous human-AI interaction loops.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 8de9ed2a-64d6-41cc-bae7-4db6015efbe7Builds on16
- Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to DesignQian Yang, Aaron Steinfeld, Carolyn P. Rosé, John ZimmermanCHI 2020 · 604 citations
- Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching AssistantQiaosi Wang, Koustuv Saha, Eric Gregori, David A. Joyner et al.CHI 2021 · 115 citations
- Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI ChallengesQiaosi Wang, Michael Madaio, Shaun K. Kane, Shivani Kapania et al.CHI 2023 · 90 citations
- Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User ExperienceQ. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman VaughanCHI 2023 · 81 citations
- Eliciting and Analysing Users' Envisioned Dialogues with Perfect Voice AssistantsSarah Theres Völkel, Daniel Buschek, Malin Eiband, Benjamin R. Cowan et al.CHI 2021 · 75 citations
Related papers
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent SystemsXuanming Zhang, Yuxuan Chen, Samuel (Min-Hsuan) Yeh, Sharon LiNeurIPS 2025 · 14 citations
- Theory of Mind in Large Language Models: Assessment and EnhancementRuirui Chen, Weifeng Jiang, Chengwei Qin, Cheston TanACL 2025
- ToM-SSI: Evaluating Theory of Mind in Situated Social InteractionsMatteo Bortoletto, Constantin Ruhdorfer, Andreas BullingEMNLP 2025
- COKE: A Cognitive Knowledge Graph for Machine Theory of MindJincenzi Wu, Zhuang Chen, Jiawen Deng, Sahand Sabour et al.ACL 2024
- Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language ModelsChani Jung, Dongkwan Kim, Jiho Jin, Jiseon Kim et al.EMNLP 2024 · 2 citations
