Lune

ACL Findings2021

Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions

Nam Do, Ellie Pavlick

Abstract

Previous studies have argued that pre-trained language models encode commonsense relational knowledge (e.g. that apples are edible). However, simultaneous work has revealed that such models are often insensitive to context, even ignoring overt contextual cues such as negations. In this paper, we investigate whether masked language models (the BERT family) can move beyond naive associative biases (e.g., apple → edible) when the context warrants (e.g. ranking inedible higher when presented with the information that the apple is rotten). We introduce the WINOVENTI procedure, which adversarially exploits generic associations in masked language models to create model-specific Winograd-style entailment schemas. Using our constructed WINOVENTI challenges set of over 2, 000 schemas, we show that language models in the BERT family experience a steep drop in performance on prompts that require them to pick answers which require reasoning about context (e.g., from 89.8% to 18.4% for BERT LARGE ). We present evidence that language models exhibit different associative biases, suggesting a need for future work in developing and analyzing frameworks similar to WINOVENTI that are tuned to model-specific weaknesses.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Adversarial NLI: A New Benchmark for Natural Language Understanding cites Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
  • Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? cites CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
  • DiscoSense: Commonsense Reasoning with Discourse Connectives cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • DiscoSense: Commonsense Reasoning with Discourse Connectives cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Knowledge Enhanced Reflection Generation for Counseling DialoguesSiqi Shen, Verónica Pérez-Rosas, Charles Welch, Soujanya Poria et al.ACL 2022 · 32 citations
  2. Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense KnowledgeJiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng et al.ACL 2023 · 23 citations
  3. CRoW: Benchmarking Commonsense Reasoning in Real-World TasksMete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva et al.EMNLP 2023 · 3 citations
  4. Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?Ning Bian, Xianpei Han, Hongyu Lin, Yaojie Lu et al.ACL 2024 · 1 citation
  5. DiscoSense: Commonsense Reasoning with Discourse ConnectivesPrajjwal Bhargava, Vincent NgEMNLP 2022 · 1 citation

Builds on6

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  3. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
  4. Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
  5. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations

Related papers

  1. The Sensitivity of Language Models and Humans to Winograd Schema PerturbationsMostafa Abdou, Vinit Ravishankar, Maria Barrett, Yonatan Belinkov et al.ACL 2020 · 1 citation
  2. Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd SchemaYanai Elazar, Hongming Zhang, Yoav Goldberg, Dan RothEMNLP 2021 · 25 citations
  3. WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema ChallengeHongming Zhang, Xinran Zhao, Yangqiu SongACL 2020 · 33 citations
  4. WinoLogic: A Zero-Shot Logic-based Diagnostic Dataset for Winograd Schema ChallengeWeinan He, Canming Huang, Yongmei Liu, Xiaodan ZhuEMNLP 2021 · 7 citations
  5. Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference ResolutionDenis Emelin, Rico SennrichEMNLP 2021 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines