SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions
Bufang Yang, Yunqi Guo, Lilin Xu, Zhenyu Yan, Hongkai Chen, Guoliang Xing, Xiaofan Fred Jiang
Abstract
Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants from Large-Scale Simulated Persona Interactions cites ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions
- After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants from Large-Scale Simulated Persona Interactions cites Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies
- After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants from Large-Scale Simulated Persona Interactions cites Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults
- ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions cites Visual Instruction Tuning
- ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies cites Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults
- Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Aligning LLM Agents by Learning Latent Preference from User Edits cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers8
- ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory PerceptionsBufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu et al.NeurIPS 2025 · 68 citations
- Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User StudiesZiyi Xuan, Yiwen Wu, Xuhai Xu, Vinod Namboodiri et al.UbiComp 2026 · 2 citations
- Surfacing and Applying Meaning: Supporting Hermeneutical Autonomy for LGBTQ+ People in TaiwanYi-Tong Chen, En-Kai Chang, Nanyi Bi, Nitesh GoyalCHI 2026 · 1 citation
- Sensing What Surveys Miss: Understanding and Personalizing Proactive LLM Support by User ModelingAilin Liu, Yesmine Karoui, Fiona Draxler, Frauke Kreuter et al.CHI 2026 · 1 citation
- After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants from Large-Scale Simulated Persona InteractionsZiyi Xuan, Yiwen Wu, Zhaoyang Yan, Vinod Namboodiri et al.UbiComp 2026
Builds on17
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older AdultsZiqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers et al.UbiComp 2024 · 116 citations
- Aligning LLM Agents by Learning Latent Preference from User EditsGe Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro et al.NeurIPS 2024 · 102 citations
Related papers
- Pro 2 Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural TasksLilin Xu, Bufang Yang, Siyang Jiang, Kaiwei Liu et al.UbiComp 2026
- Satori 悟り: Towards Proactive AR Assistant with Belief-Desire-Intention User ModelingChenyi Li, Guande Wu, Gromit Yeuk-Yin Chan, Dishita G. Turakhia et al.CHI 2025 · 49 citations
- InteractGuide: LLM-Enhanced Multimodal Reasoning for User-Centric Interaction Recommendations in AR-HRI AuthoringYunqiang Pei, Hongrong Yang, Kaiyue Zhang, Guoqing Wang et al.ACM MM 2025 · 1 citation
- Need Help? Designing Proactive AI Assistants for ProgrammingValerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar et al.CHI 2025 · 23 citations
- PMG : Personalized Multimodal Generation with Large Language ModelsXiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu et al.WWW 2024 · 40 citations
