EMNLP Findings2025
CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization
Jing Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, Yongbin Li
Abstract
Reinforcement Learning Fine-Tuning (RLFT) has achieved notable success in tasks with objectively verifiable answers (e.g., code generation, mathematical reasoning), yet struggles with open-ended subjective tasks like roleplaying dialogue. Traditional reward modeling approaches, which rely on independent samplewise scoring, face dual challenges: subjective evaluation criteria and unstable reward signals. Motivated by the insight that human evaluation inherently combines explicit criteria with implicit comparative judgments, we propose Comparative Policy Optimization (CPO). CPO redefines the reward evaluation paradigm by shifting from sample-wise scoring to comparative group-wise scoring. Building on the same principle, we introduce the Character-Arena evaluation framework, which comprises two stages: (1) Contextualized Multi-turn Roleplaying Simulation, and (2) Trajectory-level Comparative Evaluation. By operationalizing subjective scoring via objective trajectory comparisons, CharacterArena minimizes contextual bias and enables more robust and fair performance evaluation. Empirical results on Charac-terEval, CharacterBench, and CharacterArena confirm that CPO effectively mitigates reward ambiguity and leads to substantial improvements in dialogue quality.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agents cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World cites Efficient Memory Management for Large Language Model Serving with PagedAttention
Cited by top-tier papers4
- Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective AmbiguityFeiteng Fang, Dingwei Chen, Xiang Huang, Ting-En Lin et al.ACL 2026 · 3 citations
- EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal WorldJing Ye, Lu Xiang, Yaping Zhang, Chengqing ZongACL 2026 · 2 citations
- DREAM: LLM-based Dynamic Role-playing via Event-Aware Memory GraphZhihao Xiao, Mengting Li, Xintao Wang, Linfeng Li et al.KDD 2026 · 1 citation
- CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing AgentsYihong Tang, Kehai Chen, Liang Yue, Benyou Wang et al.ICML 2026
Builds on19
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyGuohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin et al.NeurIPS 2023 · 1,975 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
Related papers
- ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative RankingQiang Zhang, Boli Chen, Fanrui Zhang, Ruixue Ding et al.ICML 2026 · 10 citations
- DPO Meets PPO: Reinforced Token Optimization for RLHFHan Zhong, Zikang Shan, Guhao Feng, Wei Xiong et al.ICML 2025
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive StudyShusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye et al.ICML 2024 · 274 citations
- CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based RewardsCheng Liu, Yifei Lu, Fanghua Ye, Jian Li et al.EMNLP 2025
- R4: Nested Reasoning-Retrieval for Reward Modeling in Role-Playing AgentsRenzhi Wang, Chongqiang Wei, Zhisheng Wang, Piji LiICLR 2026
