GLOSS: Group of LLMs for Open-ended Sensemaking of Passive Sensing Data for Health and Wellbeing

The ubiquitous presence of smartphones and wearables has enabled researchers to build prediction and detection models for various health and behavior outcomes using passive sensing data from these devices. Achieving a high-level, holistic understanding of an individual's behavior and context, however, remains a significant challenge. Due to the nature of the passive sensing data, sensemaking --- the process of interpreting and extracting insights - requires both domain knowledge and technical expertise, creating barriers for different stakeholders. Existing systems designed to support sensemaking are not open-ended or cannot perform complex data triangulation. In this paper, we present a novel sensemaking system, Group of LLMs for Open-ended Sensemaking (GLOSS), for open-ended sensemaking capable of performing complex multimodal triangulation to derive insights. We demonstrate that GLOSS significantly outperforms commonly used Retrieval-Augmented Generation (RAG) technique, achieving 87.93% accuracy and 66.19% consistency compared to RAG's 29.31% accuracy and 52.85% consistency. Furthermore, we showcase the promise of GLOSS using four use cases inspired by prior and ongoing work in UbiComp and HCI communities. Finally, we discuss the potential of GLOSS, the broader implications, and the limitations of our work.