Lune

ACL Findings2021

Language Models Use Monotonicity to Assess NPI Licensing

Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, Shane Steinert-Threlkeld

Abstract

We investigate the semantic knowledge of language models (LMs), focusing on (1) whether these LMs create categories of linguistic environments based on their semantic monotonicity properties, and (2) whether these categories play a similar role in LMs as in human language understanding, using negative polarity item licensing as a case study. We introduce a series of experiments consisting of probing with diagnostic classifiers (DCs), linguistic acceptability tasks, as well as a novel DC ranking method that tightly connects the probing results to the inner workings of the LM. By applying our experimental pipeline to LMs trained on various filtered corpora, we are able to gain stronger insights into the semantic generalizations that are acquired by these models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little cites Information-Theoretic Probing with Minimum Description Length
  • Cross-Modal Taxonomic Generalization in (Vision-) Language Models cites Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs
  • Transformers in the loop: Polarity in neural models of language cites A Systematic Assessment of Syntactic Generalization in Neural Language Models
  • Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs cites Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
  • Predicting Inductive Biases of Pre-Trained Models cites Information-Theoretic Probing with Minimum Description Length
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for LittleKoustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau et al.EMNLP 2021 · 177 citations
  2. Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNsKanishka Misra, Kyle MahowaldEMNLP 2024 · 11 citations
  3. Connecting degree and polarity: An artificial language learning studyLisa Bylinina, Alexey Tikhonov, Ekaterina GarmashEMNLP 2023
  4. Cross-Modal Taxonomic Generalization in (Vision-) Language ModelsTianyang Xu, Marcelo Sandoval-Castañeda, Karen Livescu, Greg Shakhnarovich et al.ACL 2026
  5. Transformers in the loop: Polarity in neural models of languageLisa Bylinina, Alexey TikhonovACL 2022

Builds on3

  1. A Systematic Assessment of Syntactic Generalization in Neural Language ModelsJennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox et al.ACL 2020 · 124 citations
  2. Predicting Inductive Biases of Pre-Trained ModelsCharles Lovering, Rohan Jha, Tal Linzen, Ellie PavlickICLR 2021 · 70 citations
  3. Information-Theoretic Probing with Minimum Description LengthElena Voita, Ivan TitovEMNLP 2020 · 34 citations

Related papers

  1. Adjective Scale Probe: Can Language Models Encode Formal Semantics Information?Wei Liu, Ming Xiang, Nai DingAAAI 2023 · 7 citations
  2. Inspecting the concept knowledge graph encoded by modern language modelsCarlos Aspillaga, Marcelo Mendoza, Alvaro SotoACL Findings 2021 · 7 citations
  3. Implicit Representations of Grammaticality in Language ModelsYingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy et al.ACL 2026
  4. Interpretability of Language Models via Task SpacesLucas Weber, Jaap Jumelet, Elia Bruni, Dieuwke HupkesACL 2024
  5. Probing Natural Language Inference Models through Semantic FragmentsKyle Richardson, Hai Hu, Lawrence S. Moss, Ashish SabharwalAAAI 2020 · 152 citations
Dusk over the sea between two cliffs drawn in fine vertical lines