EMNLP Findings2021
Attention-based Contrastive Learning for Winograd Schemas
Abstract
Self-supervised learning has recently attracted considerable attention in the NLP community for its ability to learn discriminative features using a contrastive objective (Qu et al., 2020;Klein and Nabi, 2020). This paper investigates whether contrastive learning can be extended to Transfomer attention to tackling the Winograd Schema Challenge. To this end, we propose a novel self-supervised framework, leveraging a contrastive loss directly at the level of self-attention. Experimental analysis of our attention-based models on multiple datasets demonstrates superior commonsense reasoning capabilities. The proposed approach outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on4
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- On Identifiability in TransformersGino Brunner, Yang Liu, Damian Pascual, Oliver Richter et al.ICLR 2020 · 210 citations
- CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language UnderstandingYanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev et al.ICLR 2021 · 77 citations
- Pre-training Is (Almost) All You Need: An Application to Commonsense ReasoningAlexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot et al.ACL 2020 · 30 citations
Related papers
- Str-GCL: Structural Commonsense Driven Graph Contrastive LearningDongxiao He, Yongqi Huang, Jitao Zhao, Xiaobao Wang et al.WWW 2025 · 6 citations
- Contrastive Attention Maps for Self-supervised Co-localizationMinsong Ki, Youngjung Uh, Junsuk Choe, Hyeran ByunICCV 2021 · 11 citations
- Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language InferenceCanming Huang, Weinan He, Yongmei LiuEMNLP Findings 2021 · 8 citations
- Neural Topic Modeling based on Cycle Adversarial Training and Contrastive LearningBoyu Wang, Linhai Zhang, Deyu Zhou, Yi Cao et al.ACL Findings 2023 · 4 citations
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd SchemaYanai Elazar, Hongming Zhang, Yoav Goldberg, Dan RothEMNLP 2021 · 25 citations
