Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
Marwa Abdulhai, Ryan Cheng, Donovan Clay, Tim Althoff, Sergey Levine, Natasha Jaques
Abstract
Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue. We define three automatic metrics-prompt-to-line consistency, line-to-line consistency, and Q&A consistency-that capture different types of persona drift and validate each against human annotations. Using these metrics as reward signals, we apply multiturn reinforcement learning to fine-tune LLMs for three user roles: a patient, a student, and a social chat partner. Our method reduces inconsistency by over 55%, resulting in more coherent, faithful, and trustworthy simulated users.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization cites Training language models to follow instructions with human feedback
- Synthia: Scalable Grounded Persona Generation from Social Media Data cites Generative Agents: Interactive Simulacra of Human Behavior
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback cites Generative Agents: Interactive Simulacra of Human Behavior
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback cites Training language models to follow instructions with human feedback
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society cites Training language models to follow instructions with human feedback
- Generative Agents: Interactive Simulacra of Human Behavior cites Training language models to follow instructions with human feedback
- HumanLM: Simulating Users with State Alignment Beats Response Imitation cites Generative Agents: Interactive Simulacra of Human Behavior
- HumanLM: Simulating Users with State Alignment Beats Response Imitation cites Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
- Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies cites Training language models to follow instructions with human feedback
Cited by top-tier papers4
- Simulated Students in Tutoring Dialogues: Substance or Illusion?Alexander Scarlatos, Jaewook Lee, Simon Woodhead, Andrew LanACL 2026 · 6 citations
- Synthia: Scalable Grounded Persona Generation from Social Media DataVahid Rahimzadeh, Erfan Moosavi Monazzah, Mohammad Taher Pilehvar, Yadollah YaghoobzadehACL 2026 · 1 citation
- Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic OptimizationYihang Yao, Zhepeng Cen, Haohong Lin, Shiqi Liu et al.ICML 2026
- HumanLM: Simulating Users with State Alignment Beats Response ImitationShirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang et al.ICML 2026
Builds on24
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyGuohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin et al.NeurIPS 2023 · 1,975 citations
- Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris et al.UIST 2023 · 1,882 citations
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human FeedbackYann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang et al.NeurIPS 2023 · 948 citations
- Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject StudiesGati V. Aher, Rosa I. Arriaga, Adam Tauman KalaiICML 2023 · 651 citations
Related papers
- Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMsManon Reusens, Bart Baesens, David JurgensEMNLP Findings 2025 · 1 citation
- Can LLM Agents Maintain a Persona in Discourse?Pranav Bhandari, Nicolas Fay, Michael J. Wise, Amitava Datta et al.EMNLP 2025
- Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended GenerationJisu Shin, Juhyun Oh, Eunsu Kim, Hoyun Song et al.ACL Findings 2025 · 1 citation
- The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMsPengrui Han, Rafal Kocielnik, Peiyang Song, Ramit Debnath et al.ICML 2026 · 33 citations
- LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language ModelsMarwa Abdulhai, Isadora White, Charlie Victor Snell, Charles Sun et al.ICML 2025
