Lune

EMNLP Findings2021

Attention-based Contrastive Learning for Winograd Schemas

Tassilo Klein, Moin Nabi

Abstract

Self-supervised learning has recently attracted considerable attention in the NLP community for its ability to learn discriminative features using a contrastive objective (Qu et al., 2020;Klein and Nabi, 2020). This paper investigates whether contrastive learning can be extended to Transfomer attention to tackling the Winograd Schema Challenge. To this end, we propose a novel self-supervised framework, leveraging a contrastive loss directly at the level of self-attention. Experimental analysis of our attention-based models on multiple datasets demonstrates superior commonsense reasoning capabilities. The proposed approach outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. ConVQG: Contrastive Visual Question Generation with Multimodal GuidanceLi Mi, Syrielle Montariol, Javiera Castillo Navarro, Xianjie Dai et al.AAAI 2024 · 8 citations

Builds on4

  1. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  2. On Identifiability in TransformersGino Brunner, Yang Liu, Damian Pascual, Oliver Richter et al.ICLR 2020 · 210 citations
  3. CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language UnderstandingYanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev et al.ICLR 2021 · 77 citations
  4. Pre-training Is (Almost) All You Need: An Application to Commonsense ReasoningAlexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot et al.ACL 2020 · 30 citations

Related papers

  1. Str-GCL: Structural Commonsense Driven Graph Contrastive LearningDongxiao He, Yongqi Huang, Jitao Zhao, Xiaobao Wang et al.WWW 2025 · 6 citations
  2. Contrastive Attention Maps for Self-supervised Co-localizationMinsong Ki, Youngjung Uh, Junsuk Choe, Hyeran ByunICCV 2021 · 11 citations
  3. Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language InferenceCanming Huang, Weinan He, Yongmei LiuEMNLP Findings 2021 · 8 citations
  4. Neural Topic Modeling based on Cycle Adversarial Training and Contrastive LearningBoyu Wang, Linhai Zhang, Deyu Zhou, Yi Cao et al.ACL Findings 2023 · 4 citations
  5. Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd SchemaYanai Elazar, Hongming Zhang, Yoav Goldberg, Dan RothEMNLP 2021 · 25 citations
Dusk over the sea between two cliffs drawn in fine vertical lines