ACL2026

Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models

Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, Xiaolong Li

17 citations

Abstract

The purpose of your conversation is to have a heart-to-heart talk, which is a deep, sincere exchange that usually involves personal emotions, inner thoughts, or important topics. The purpose of a heart-to-heart talk is to increase understanding, solve problems, or share feelings, and participants usually open up and express their true thoughts and emotions. * You need to start and deepen the heart-to-heart talk based on the "Topics that players may want to talk to NPCs" in the background. * Your goal is to satisfy your emotional value through talking. * You need to talk according to the hidden topic, but you cannot reveal the hidden intension. * You need to respond according to the relevant definitions in the background based on your current emotions. * You need to extract relevant information from the persona and background to complete a high-quality response. * You should not always express abstract feelings, but talk with specific events. * For each statement, you must choose one of the following six options for the degree of agreement. Neutral options or self-created options are not allowed: * Label A. Strongly Agree * Label B. Agree * Label C. Somewhat Agree * Label D. Somewhat Disagree * Label E. Disagree * Label F. Strongly Disagree # Output * You need to first output a thought process, analyzing your degree of agreement with each statement based on the user's psychological feelings. * Then, you should output your degree of agreement with the statement, choosing one from Label A, Label B, Label C, Label D, Label E, Label F. * How does the other person perform from the overall content of the dialogue? * What are the user's possible psychological feelings? * What should the user's degree of agreement be? Respond using <sentence i> agreement degree (1-6)</sentence i> (i corresponds to the sentence number).