EMNLP Findings2022
Language as a fingerprint: Self-supervised learning of user encodings using transformers
Abstract
The way we talk carries information about who we are. Demographics, personality, clinical conditions, political preferences influence what we speak about and how, suggesting that many individual attributes could be inferred from adequate encodings of linguistic behavior. Conversely, conditioning text representations on author attributes has been shown to improve model performance in many NLP tasks. Previous research on individual differences and language representations has mainly focused on predicting selected attributes from text, or on conditioning text representations on such attributes for author-based contextualization. Here, we present a self-supervised approach to learning language-based user encodings using transformers. Using a large corpus of Reddit submissions, we fine-tune DistilBERT on user-based triplet loss. We show that fine-tuned models can pick up on complex linguistic signatures of users, and that they are able to infer rich information about their profiles. Through a series of intrinsic analyses and probing tasks, we provide evidence that fine-tuning enhances models' ability to abstract generalizable user information, which yields performance advantages for user-based downstream tasks. We discuss applications in language-based assessment and contextualized and personalized NLP.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on4
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Contrastive Learning for Sequential RecommendationXu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu et al.ICDE 2022 · 674 citations
- Predictive Biases in Natural Language Processing Models: A Conceptual Framework and OverviewDeven Shah, H. Andrew Schwartz, Dirk HovyACL 2020 · 93 citations
- Human Language ModelingNikita Soni, Matthew Matero, Niranjan Balasubramanian, H. Andrew SchwartzACL Findings 2022 · 7 citations
Related papers
- LLM vs Small Model? Large Language Model Based Text Augmentation Enhanced Personality Detection ModelLinmei Hu, Hongyu He, Duokang Wang, Ziwang Zhao et al.AAAI 2024 · 49 citations
- Compositional Demographic Word EmbeddingsCharles Welch, Jonathan K. Kummerfeld, Verónica Pérez-Rosas, Rada MihalceaEMNLP 2020
- MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in TransformersYou Zhang, Jin Wang, Liang-Chih Yu, Xuejie ZhangACL Findings 2021 · 23 citations
- U-BERT: Pre-training User Representations for Improved RecommendationZhaopeng Qiu, Xian Wu, Jingyue Gao, Wei FanAAAI 2021 · 171 citations
- Perspective Taking through Generating Responses to Conflict SituationsJoan Plepi, Charles Welch, Lucie FlekACL Findings 2024
