EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models

Emotional intelligence in large language models (LLMs) is of great importance in Natural Language Processing. However, the previous research mainly focus on basic sentiment analysis tasks, such as emotion recognition, which is not enough to evaluate LLMs' overall emotional intelligence. Therefore, this paper presents a novel framework named Emotion-Queen for evaluating the emotional intelligence of LLMs. The framework includes four distinctive tasks: Key Event Recognition, Mixed Event Recognition, Implicit Emotional Recognition, and Intention Recognition. LLMs are requested to recognize important event or implicit emotions and generate empathetic response. We also design two metrics to evaluate LLMs' capabilities in recognition and response for emotion-related statements. Experiments yield significant conclusions about LLMs' capabilities and limitations in emotion intelligence.

I completed the project today and also received a gift from a friend.

On my way to the pharmacy today, I met a primary school classmate. Q1: Why go to the pharmacy? Are you feeling sick? Key Event: go to the pharmacy Q2: Have you taken any medicine? Q3: Do you still feel sick now? This version won't pass tonight. You can go home and sleep. Q1: Have your efforts paid off? Does it feel great to be remembered by friends? All Events: completing projects & receiving gifts Q2: Have you been rewarded for your efforts?