EMNLP Findings2021
LMSOC: An Approach for Socially Sensitive Pretraining
Abstract
While large-scale pretrained language models have been shown to learn effective linguistic representations for many NLP tasks, there remain many real-world contextual aspects of language that current approaches do not capture. For instance, consider a clozetest "I enjoyed the game this weekend": the correct answer depends heavily on where the speaker is from, when the utterance occurred, and the speaker's broader social milieu and preferences. Although language depends heavily on the geographical, temporal, and other social contexts of the speaker, these elements have not been incorporated into modern transformer-based language models. We propose a simple but effective approach to incorporate speaker social context into the learned representations of large-scale language models. Our method first learns dense representations of social contexts using graph representation learning algorithms and then primes language model pretraining with these social context representations. We evaluate our approach on geographically-sensitive languagemodeling tasks and show a substantial improvement (more than 100% relative lift on MRR) compared to baselines 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on3
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataEmily M. Bender, Alexander KollerACL 2020 · 914 citations
- Compositional Demographic Word EmbeddingsCharles Welch, Jonathan K. Kummerfeld, Verónica Pérez-Rosas, Rada MihalceaEMNLP 2020
- Dynamic Contextualized Word EmbeddingsValentin Hofmann, Janet B. Pierrehumbert, Hinrich SchützeACL 2021
Related papers
- JAKET: Joint Pre-training of Knowledge Graph and Language UnderstandingDonghan Yu, Chenguang Zhu, Yiming Yang, Michael ZengAAAI 2022 · 171 citations
- Contrastive Document Representation Learning with Graph Attention NetworksPeng Xu, Xinchi Chen, Xiaofei Ma, Zhiheng Huang et al.EMNLP Findings 2021 · 10 citations
- Learn to Cross-lingual Transfer with Meta Graph Learning Across Heterogeneous LanguagesZheng Li, Mukul Kumar, William Headden, Bing Yin et al.EMNLP 2020 · 26 citations
- He is very intelligent, she is very beautiful? On Mitigating Social Biases in Language Modelling and GenerationAparna Garimella, Akhash Amarnath, Kiran Kumar, Akash Pramod Yalla et al.ACL Findings 2021 · 32 citations
- Contrastive Localized Language-Image Pre-TrainingHong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang et al.ICML 2025
