Lune

EMNLP Findings2023

RealBehavior: A Framework for Faithfully Characterizing Foundation Models' Human-like Behavior Mechanisms

Enyu Zhou, Rui Zheng, Zhiheng Xi, Songyang Gao, Xiaoran Fan, Zichu Fei, Jingting Ye, Tao Gui, Qi Zhang, Xuanjing Huang

Abstract

Reports of human-like behaviors in foundation models are growing, with psychological theories providing enduring tools to investigate these behaviors. However, current research tends to directly apply these human-oriented tools without verifying the faithfulness of their outcomes. In this paper, we introduce a framework, RealBehavior, which is designed to characterize the humanoid behaviors of models faithfully. Beyond simply measuring behaviors, our framework assesses the faithfulness of results based on reproducibility, internal and external consistency, and generalizability. Our findings suggest that a simple application of psychological tools cannot faithfully characterize all human-like behaviors. Moreover, we discuss the impacts of aligning models with human and social values, arguing for the necessity of diversifying alignment objectives to prevent the creation of models with restricted characteristics.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Evaluating Psychological Safety of Large Language Models cites Training language models to follow instructions with human feedback
  • RMB: Comprehensively benchmarking reward models in LLM alignment cites Training language models to follow instructions with human feedback
  • Training Socially Aligned Language Models on Simulated Social Interactions cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. RMB: Comprehensively benchmarking reward models in LLM alignmentEnyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi et al.ICLR 2025

Builds on3

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Training Socially Aligned Language Models on Simulated Social InteractionsRuibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang et al.ICLR 2024 · 97 citations
  3. Evaluating Psychological Safety of Large Language ModelsXingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty et al.EMNLP 2024 · 14 citations

Related papers

  1. Pressure Reveals Character: Behavioural Alignment Evaluation at DepthNora Petrova, John BurdenICML 2026
  2. HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive PatternsXintao Wang, Jian Yang, Weiyuan Li, Rui Xie et al.ACL 2026
  3. Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow PreferencesJoshua Ashkinaze, Hua Shen, Sai Avula, Eric Gilbert et al.NeurIPS 2025 · 7 citations
  4. A Computational Framework for Evaluating Human-likeness in LLMs' Open-ended Human BehaviorsYuxuan Lei, Jianxun Lian, Defu Lian, Jincenzi Wu et al.ICML 2026
  5. Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended GenerationJisu Shin, Juhyun Oh, Eunsu Kim, Hoyun Song et al.ACL Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines