ACL Findings2026
Understanding Generalization in Role-Playing Models via Information Theory
Abstract
Role-playing models (RPMs) are widely used in real-world applications but underperform when deployed in the wild. This degradation can be attributed to distribution shifts, including user, character, and dialogue compositional shifts. Existing methods like LLM-as-a-judge fall short in providing a fine-grained diagnosis of how these shifts affect RPM generalization, and thus there lack formal frameworks to characterize RPM generalization behaviors. To bridge these gaps, we introduce an information-theoretic metric, named reasoning-based effective mutual information difference (R-EMID), to measure RPM performance degradation in an interpretable way. We also derive an upper bound on R-EMID to predict the worst-case generalization performance of RPMs and theoretically reveal how various shifts contribute to the RPM performance degradation. Moreover, we propose a coevolving reinforcement learning framework to adaptively model the connection among user, character, and dialogue context and thus enhance the estimation of dialogue response generation probability, which is critical for calculating R-EMID. Finally, we evaluate the generalization performance of various RPMs using R-EMID, finding that user shift poses the highest risk among all shifts and reinforcement learning is the most effective approach for enhancing RPM generalization. Code and data are available at https://github.com/ AlibabaResearch/DAMO-ConvAI/tree/ main/RPM-Generalization.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SocialBench: Sociality Evaluation of Role-Playing Conversational Agents cites Character-LLM: A Trainable Agent for Role-Playing
- CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agents cites SocialBench: Sociality Evaluation of Role-Playing Conversational Agents
- CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agents cites Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning
- Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning cites SocialBench: Sociality Evaluation of Role-Playing Conversational Agents
- Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning cites Character-LLM: A Trainable Agent for Role-Playing
Cited by top-tier papers1
Builds on12
- Character-LLM: A Trainable Agent for Role-PlayingYunfan Shao, Linyang Li, Junqi Dai, Xipeng QiuEMNLP 2023 · 97 citations
- LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations
- Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with CharactersNuo Chen, Yan Wang, Haiyun Jiang, Deng Cai et al.EMNLP Findings 2023 · 28 citations
- SocialBench: Sociality Evaluation of Role-Playing Conversational AgentsHongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu et al.ACL Findings 2024 · 16 citations
- Thinking in Character: Advancing Role-Playing Agents with Role-Aware ReasoningYihong Tang, Kehai Chen, Muyun Yang, Zheng-Yu Niu et al.NeurIPS 2025 · 16 citations
Related papers
- Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic ApproachChangdae Oh, Zhen Fang, Shawn Im, Xuefeng Du et al.ICML 2025
- CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent EvaluationQuan Tu, Shilong Fan, Zihang Tian, Tianhao Shen et al.ACL 2024
- CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based RewardsCheng Liu, Yifei Lu, Fanghua Ye, Jian Li et al.EMNLP 2025
- R4: Nested Reasoning-Retrieval for Reward Modeling in Role-Playing AgentsRenzhi Wang, Chongqiang Wei, Zhisheng Wang, Piji LiICLR 2026
- Consistently Simulating Human Personas with Multi-Turn Reinforcement LearningMarwa Abdulhai, Ryan Cheng, Donovan Clay, Tim Althoff et al.NeurIPS 2025 · 51 citations
