ACL Findings2025
Systematic Evaluation of Auto-Encoding and Large Language Model Representations for Capturing Author States and Traits
Khushboo Singh, Vasudha Varadarajan, Adithya V. Ganesan, August Håkan Nilsson, Nikita Soni, Syeda Mahwish, Pranav Chitale, Ryan L. Boyd, Lyle H. Ungar, Richard N. Rosenthal, H. Andrew Schwartz
Abstract
Large Language Models (LLMs) are increasingly used in human-centered applications, yet their ability to model diverse psychological constructs is not well understood. In this study, we systematically evaluate a range of Transformer-LMs to predict psychological variables across five major dimensions: affect, substance use, mental health, sociodemographics, and personality. Analyses span three temporal levelsshort daily text responses about current affect, text aggregated over two-weeks, and user-level text collected over two years-allowing us to examine how each model's strengths align with the underlying stability of different constructs. The findings show that mental health signals emerge as the most accurately predicted dimensions (r ≈ 0.6) across all temporal scales. At the daily scale, smaller models like DeBERTa and HaRT often performed better, whereas, at longer scales or with greater context, larger model like Llama3-8B performed the best. Also, aggregating text over the entire study period yielded stronger correlations for outcomes, such as age and income. Overall, these results suggest the importance of selecting appropriate model architectures and temporal aggregation techniques based on the stability and nature of the target variable.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? cites Deberta: decoding-Enhanced Bert with Disentangled Attention
- Human Language Modeling cites Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention
- From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP cites Human Language Modeling
Cited by top-tier papers1
Builds on6
- Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
- Hierarchical Modeling for User Personality Prediction: The Role of Message-Level AttentionVeronica E. Lynn, Niranjan Balasubramanian, H. Andrew SchwartzACL 2020 · 72 citations
- Measuring Forecasting Skill from TextShi Zong, Alan Ritter, Eduard H. HovyACL 2020 · 21 citations
- Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning?Muhammad Reza Qorib, Geonsik Moon, Hwee Tou NgACL Findings 2024 · 15 citations
- Human Language ModelingNikita Soni, Matthew Matero, Niranjan Balasubramanian, H. Andrew SchwartzACL Findings 2022 · 7 citations
Related papers
- From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological ProfilersYi-Fei Liu, Yi-Long Lu, Di He, Hang ZhangICLR 2026 · 6 citations
- On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMsJen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam et al.ICLR 2024 · 85 citations
- On the Reliability of Psychological Scales on Large Language ModelsJen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li et al.EMNLP 2024 · 6 citations
- Assessment and manipulation of latent constructs in pre-trained language models using psychometric scalesMaor Reuben, Ortal Slobodin, Idan-Chaim Cohen, Aviad Elyashar et al.ACL 2025 · 7 citations
- Manipulating the Perceived Personality Traits of Language ModelsGraham Caron, Shashank SrivastavaEMNLP Findings 2023 · 20 citations
