EMNLP Findings2021
MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection
Matthew Matero, Nikita Soni, Niranjan Balasubramanian, H. Andrew Schwartz
Abstract
Much of natural language processing is focused on leveraging large capacity language models, typically trained over single messages with a task of predicting one or more tokens. However, modeling human language at higher-levels of context (i.e., sequences of messages) is underexplored. In stance detection and other social media tasks where the goal is to predict an attribute of a message, we have contextual data that is loosely semantically connected by authorship. Here, we introduce Message-Level Transformer (MeLT) -a hierarchical messageencoder pre-trained over Twitter and applied to the task of stance prediction. We focus on stance prediction as a task benefiting from knowing the context of the message (i.e., the sequence of previous messages). The model is trained using a variant of masked-language modeling; where instead of predicting tokens, it seeks to generate an entire masked (aggregated) message vector via reconstruction loss. We find that applying this pre-trained masked messagelevel transformer to the downstream task of stance detection achieves F1 performance of 67%.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
- Stanceosaurus: Classifying Stance Towards Multicultural MisinformationJonathan Zheng, Ashutosh Baheti, Tarek Naous, Wei Xu et al.EMNLP 2022 · 6 citations
- From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLPAdithya V. Ganesan, Vasudha Varadarajan, Oscar N. E. Kjell, Whitney Ringwald et al.ACL 2026 · 3 citations
Builds on4
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- Hierarchical Modeling for User Personality Prediction: The Role of Message-Level AttentionVeronica E. Lynn, Niranjan Balasubramanian, H. Andrew SchwartzACL 2020 · 72 citations
- Coupled Hierarchical Transformer for Stance-Aware Rumor Verification in Social Media ConversationsJianfei Yu, Jing Jiang, Ling Min Serena Khoo, Hai Leong Chieu et al.EMNLP 2020 · 49 citations
Related papers
- Human Language ModelingNikita Soni, Matthew Matero, Niranjan Balasubramanian, H. Andrew SchwartzACL Findings 2022 · 7 citations
- Multimodal Multi-turn Conversation Stance Detection: A Challenge Dataset and Effective ModelFuqiang Niu, Zebang Cheng, Xianghua Fu, Xiaojiang Peng et al.ACM MM 2024 · 13 citations
- Stanceformer: Target-Aware Transformer for Stance DetectionKrishna Garg, Cornelia CarageaEMNLP Findings 2024 · 9 citations
- Chain-of-Thought Embeddings for Stance Detection on Social MediaJoseph Gatto, Omar Sharif, Sarah PreumEMNLP Findings 2023 · 18 citations
- Learning Multiscale Transformer Models for Sequence GenerationBei Li, Tong Zheng, Yi Jing, Chengbo Jiao et al.ICML 2022 · 15 citations
