Locate and Explain: Joint Multimodal Emotion Cause Extraction and Summarization in Conversation

Multimodal emotion cause analysis in conversation aims to identify the causes of emotions by leveraging multimodal information. Existing studies mainly formulate this problem as either utterance-level emotion cause extraction, which provides clear cause localization but limited explanation, or multimodal emotion cause generation, which offers fine-grained explanations but lacks explicit traceability to source utterances. Moreover, existing datasets rely heavily on human judgment and lack well-defined structured theoretical guidance, leading to subjective and inconsistent annotations. To address these issues, we introduce joint Multimodal Emotion Cause Extraction and Summarization in conversation (MECES), a new task that simultaneously extracts emotion cause utterances and generates cause summaries, enabling both precise localization and interpretable explanations of emotion cause. We further construct a MECES dataset guided by the Activating events-Beliefs-Consequences theory from psychology. This dataset consists of 5,787 emotion utterances annotated with causes, comprising 12,231 emotion-cause pairs and 6,040 cause summaries. We also propose an effective endto-end joint learning approach for MECES task, establishing strong benchmark results for this newly introduced task and dataset. * Corresponding author. ( U1,U2 , "Chuan Bai handed Guang Shi a gift, and Guang Shi didn't expect to receive one too.") Guang Shi： "I got a gift too!" Guang Shi： "Thank you, teacher!" Guang Shi： "Teacher, this gift of yours is really practical..." U1 U2 U3 U4 U5 Chuan Bai: "Here, this is from the teacher for you." Chuan Bai: "Achieving this rank is just the first step you've taken. learning Go is endless. Go back and read it carefully!" Neutral Surprise Happy Sad Neutral Why does Guang Shi feel Surprised，Happy and Sad in this dialog? ( U4 , "Guang Shi discovered the gift his teacher gave him was a Go book, he didn't expect the teacher's gift would be for him to study again") ( U1,U2 , "Chuan Bai handed Guang Shi a gift, and Guang Shi didn't expect to receive one too.")