EmoDETective: Detecting, Exploring, and Thinking Emotional Cause in Videos

There exists an affective gap between the video content and the emotions that the video creator hopes to evoke in viewers. Existing methods for video emotional content analysis attempt to learn emotion-related features directly or enhance the discrimination of models, but lack emotional cause descriptions, limiting their interpretability and the model's reasoning capabilities. In this work, we introduce EmoCause, the first large-scale video emotional dataset with multi-attribute, multi-split emotional cause descriptions. EmoCause builds upon existing datasets and is divided into 14K video splits, with over 294K emotional cause descriptions. Inspired by psychology and video prior knowledge, each video split is linked to four primary emotional cause attributes: audio, visuals, content, and shot, further divided into 12 sub-attributes, with each cause including a fact and analysis. Then, we merge all the causes into an emotional chain-of-thought for the entire video to enhance the reasoning process. Furthermore, we develop a multimodal large language model (MLLM) for video emotional content analysis, EmoDETective. EmoDETective performs training on EmoCause using progressive learning, which includes Detecting cause fact, Exploring cause analysis, and Thinking with complete reasoning. Experimental results show that our approach surpasses the existing MLLMs baseline and outperforms state-of-the-art methods, demonstrating superior emotional analysis capabilities. Ablation experiments indicate improvements from both the proposed dataset and training strategy. Code and datasets: https://github.com/Listever/EmoDETective/