ACL Findings2024
EmpathicStories++: A Multimodal Dataset for Empathy Towards Personal Experiences
Jocelyn Shen, Yubin Kim, Mohit Hulse, Wazeer Zulfikar, Sharifa Alghowinem, Cynthia Breazeal, Hae Won Park
Abstract
Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset (https://mitmedialab.github.io/empathic-stories-multimodal/) containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants' homes, as participants engage in natural, empathic storytelling interactions with AI agents. We then introduce a novel task of predicting individuals' empathy toward others' stories based on their personal experiences, evaluated in two contexts: participants' own personal shared story context and their reflections on stories they read. We benchmark this task using state-of-the-art models to pave the way for future improvements in contextualized and longitudinal empathy modeling. Our work provides a valuable resource for further research in developing empathetic AI systems and understanding the intricacies of human empathy within genuine, real-world settings.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMs cites A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
- Empathy Prediction from Diverse Perspectives cites A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Cited by top-tier papers2
Builds on7
- Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMsMaarten Sap, Ronan Le Bras, Daniel Fried, Yejin ChoiEMNLP 2022 · 92 citations
- M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue DatabaseJinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu et al.ACL 2022 · 88 citations
- MEmoR: A Dataset for Multimodal Emotion Reasoning in VideosGuangyao Shen, Xin Wang, Xuguang Duan, Hongzhi Li et al.ACM MM 2020 · 38 citations
- A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health SupportAshish Sharma, Adam S. Miner, David C. Atkins, Tim AlthoffEMNLP 2020 · 21 citations
- MEDIC: A Multimodal Empathy Dataset in CounselingZhouan Zhu, Chenguang Li, Jicai Pan, Xin Li et al.ACM MM 2023 · 9 citations
Related papers
- Modeling Empathic Similarity in Personal NarrativesJocelyn Shen, Maarten Sap, Pedro Colon-Hernandez, Hae Park et al.EMNLP 2023 · 5 citations
- STICKERCONV: Generating Multimodal Empathetic Responses from ScratchYiqun Zhang, Fanheng Kong, Peidong Wang, Shuang Sun et al.ACL 2024
- Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based BenchmarkHan Zhang, Zixiang Meng, Meng Luo, Hong Han et al.WWW 2025 · 25 citations
- ECC: An Emotion-Cause Conversation Dataset for Empathy ResponseYuanyuan He, Yongsen Pan, Wei Li, Jiali You et al.EMNLP 2025
- EmpMFF: A Multi-factor Sequence Fusion Framework for Empathetic Response GenerationXiaobing Pang, Yequan Wang, Siqi Fan, Lisi Chen et al.WWW 2023 · 13 citations
