Lune

EMNLP Findings2025

CharacterCraft: Bridging the Literature-Reality Dialogue Gap for Practical Role-Playing Agents

Xuyan Yin, Xinran Yang, Zihao Li, Lixin Zou, Chenliang Li

Abstract

Recent advancements in large language models (LLMs) have given rise to the emergence of role-playing agents (RPAs). The development of high-quality dialogue datasets is critical for advancing RPAs. However, existing datasets have two main issues: (1) the bias between query distributions and real-world user language usage, and (2) the challenge of ensuring responses accurately reflect character traits. To address these issues, we propose Character-Craft, a novel framework designed for practical RPAs, comprising a tailored Chinese roleplaying dataset and a robust evaluation method. First, we develop a specialized model for Chinese dialogue extraction, achieving state-of-theart performance. Using this model, we then extract a large amount of character dialogue from novels, ensuring high data quality (issue 2). To mitigate the literature-reality dialogue bias in extracted dialogue (issue 1), we introduce an iterative augmentation-reconstruction method, which revises queries to better align with common language usage. Additionally, we propose a context-aware memory retrieval module for fine-grained alignment with the character and introduce a reference-guided LLM-as-a-judge evaluation method for more reliable assessments by comparing their responses to source material dialogues. Our automated pipeline produces a large-scale high-quality Chinese role-playing dataset with 21,392 samples and 121,418 utterances. The experimental results demonstrate the effectiveness of our framework and reveal the limitations of existing RPAs when faced with diverse scenes. Our repository is at https://github.com/yin214/CharacterCraft.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. Character-LLM: A Trainable Agent for Role-PlayingYunfan Shao, Linyang Li, Junqi Dai, Xipeng QiuEMNLP 2023 · 97 citations
  2. NaturalConv: A Chinese Dialogue Dataset Towards Multi-turn Topic-driven ConversationXiaoyang Wang, Chen Li, Jianqiao Zhao, Dong YuAAAI 2021 · 54 citations
  3. AugESC: Dialogue Augmentation with Large Language Models for Emotional Support ConversationChujie Zheng, Sahand Sabour, Jiaxin Wen, Zheng Zhang et al.ACL Findings 2023 · 49 citations
  4. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgeDawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi et al.EMNLP 2025 · 37 citations
  5. Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with CharactersNuo Chen, Yan Wang, Haiyun Jiang, Deng Cai et al.EMNLP Findings 2023 · 28 citations

Related papers

  1. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent EvaluationQuan Tu, Shilong Fan, Zihang Tian, Tianhao Shen et al.ACL 2024
  2. Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent FrameworkBohao Yang, Dong Liu, Chenghao Xiao, Kun Zhao et al.EMNLP Findings 2025 · 2 citations
  3. Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional WorksXinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin et al.EMNLP 2024 · 2 citations
  4. R4: Nested Reasoning-Retrieval for Reward Modeling in Role-Playing AgentsRenzhi Wang, Chongqiang Wei, Zhisheng Wang, Piji LiICLR 2026
  5. CharacterBench: Benchmarking Character Customization of Large Language ModelsJinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi et al.AAAI 2025 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines