ACL Findings2022
Human Language Modeling
Nikita Soni, Matthew Matero, Niranjan Balasubramanian, H. Andrew Schwartz
Abstract
Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently.
Here, we propose human language modeling (HuLM), a hierarchical extension to the language modeling problem whereby a humanlevel exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for the HULM task, pre-trained on approximately 100,000 social media users, and demonstrate it's effectiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning documentand user-levels: stance detection, sentiment classification, age estimation, and personality assessment.
1 Results on all tasks meet or surpass the current state-of-the-art.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection cites Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention
- From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP cites WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning
- From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP cites MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection
Cited by top-tier papers7
- Large Language Models Empowered Personalized Web AgentsHongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu et al.WWW 2025 · 62 citations
- Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and BeyondTianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng et al.ICLR 2024 · 46 citations
- WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher LearningRajath Rao, Adithya V. Ganesan, Oscar N. E. Kjell, Jonah Luby et al.ACL 2025 · 4 citations
- From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLPAdithya V. Ganesan, Vasudha Varadarajan, Oscar N. E. Kjell, Whitney Ringwald et al.ACL 2026 · 3 citations
- Towards Author-informed NLP: Mind the Social BiasInbar Pendzel, Einat MinkovEMNLP 2025
Builds on6
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
- Plug and Play Language Models: A Simple Approach to Controlled Text GenerationSumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung et al.ICLR 2020 · 1,166 citations
- Predictive Biases in Natural Language Processing Models: A Conceptual Framework and OverviewDeven Shah, H. Andrew Schwartz, Dirk HovyACL 2020 · 93 citations
- Hierarchical Modeling for User Personality Prediction: The Role of Message-Level AttentionVeronica E. Lynn, Niranjan Balasubramanian, H. Andrew SchwartzACL 2020 · 72 citations
- MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance DetectionMatthew Matero, Nikita Soni, Niranjan Balasubramanian, H. Andrew SchwartzEMNLP Findings 2021 · 17 citations
Related papers
- Exciting Mood Changes: A Time-aware Hierarchical Transformer for Change Detection ModellingAnthony Hills, Talia Tseriotou, Xenia Miscouridou, Adam Tsakalidis et al.ACL Findings 2024
- Systematic Evaluation of Auto-Encoding and Large Language Model Representations for Capturing Author States and TraitsKhushboo Singh, Vasudha Varadarajan, Adithya V. Ganesan, August Håkan Nilsson et al.ACL Findings 2025
- LLM vs Small Model? Large Language Model Based Text Augmentation Enhanced Personality Detection ModelLinmei Hu, Hongyu He, Duokang Wang, Ziwang Zhao et al.AAAI 2024 · 49 citations
- From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language UnderstandingLi Sun, Florian Luisier, Kayhan Batmanghelich, Dinei A. F. Florêncio et al.ACL 2023
- Leveraging the Dual Capabilities of LLM: LLM-Enhanced Text Mapping Model for Personality DetectionWeihong Bi, Feifei Kou, Lei Shi, Yawen Li et al.AAAI 2025 · 3 citations
