VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models

Four young girls are sitting on the floor in front of a gray sofa with multiple patterned cushions, all gazing forward, seemingly at a screen. The girl on the left has brown, wavy, long hair, big almond-shaped brown eyes, thick brown eyebrows, a small nose with a high bridge, and thick pink lips. She is wearing a gray long-sleeved shirt and bright pink pants. Initially, her mouth is closed, but around mid of the video, she opens her mouth in a slight gasp. The girl in the center has brown, curly, long hair, big almond-shaped brown eyes, thick brown eyebrows, a small nose with a high bridge, and thick pink lips. She wears a light pink tank top and gray pants and has bracelets on both her wrists. She sits with her arms clasped around her knees. She is pleased with TV show, that is indicated by her smile. The girl to the immediate right has blonde, straight, long hair, big almond-shaped blue eyes, narrow blonde eyebrows, a small nose with a high bridge, and thick pink lips. She wears a light pink shirt and a white pajama bottom, as well as a bracelet on her left wrist, and is holding a small piece of popcorn. At first her mouth is closed, then she puts the popcorn into her mouth, her expression briefly shows concern; from the mid, she seems shocked by something in the screen at throw out the popcorn, and the expression turns fear. The fourth girl sits on the far right, with blonde, straight, long hair, big almond-shaped blue eyes, narrow blonde eyebrows, a small nose with a high bridge, and thick pink lips. She is wearing a light purple shirt with white and pink polka dot pajama pants. From the mid of video, she's chewing popcorn with her mouth closed, and again she is seen eating popcorn at the end. They all have fair, smooth skin and are focused forward. The lighting is from above with white light. The camera angle is front-facing and there is no camera movement. Figure 1: Selected examples of inputs and outputs obtained from VidEmo. Apart from providing toolkits for basic attribute perception and expression analysis (top), VidEmo extends the cognitive capacity and is able to generate fine-grained emotional captions with explainable rationale (bottom).