Lune

ACL Findings2025

Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events

James A. Michaelov, Reeka Estacio, Zhien Zhang, Ben Bergen

Abstract

Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show that despite the results of previous work, language models' ability to do this is far from robust. In fact, under certain conditions, all models tested-including Llama 3, Gemma 2, and Mistral NeMo-perform at worse-than-chance level, assigning higher probabilities to impossible sentences such as 'the car was given a parking ticket by the brake' than to merely unlikely sentences such as 'the car was given a parking ticket by the explorer'.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • Few-shot Learning with Multilingual Generative Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and ScaleJames A. Michaelov, Roger P. Levy, Benjamin BergenNeurIPS 2025 · 15 citations

Builds on11

  1. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  2. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  3. The Pitfalls of Simplicity Bias in Neural NetworksHarshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain et al.NeurIPS 2020 · 503 citations
  4. Few-shot Learning with Multilingual Generative Language ModelsXi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang et al.EMNLP 2022 · 113 citations
  5. A Case Study of the Shortcut Effects in Visual Commonsense ReasoningKeren Ye, Adriana KovashkaAAAI 2021 · 47 citations

Related papers

  1. Rare Event Analysis of Large Language ModelsJake McAllister Dorman, Edward Gillman, Dominic C Rose, Jamie Mair et al.ICML 2026
  2. Evaluating Distributional Distortion in Neural Language ModelingBenjamin LeBrun, Alessandro Sordoni, Timothy J. O'DonnellICLR 2022 · 26 citations
  3. Language Model Probabilities are Not Calibrated in Numeric ContextsCharles Lovering, Michael Krumdick, Viet Dac Lai, Varshini Reddy et al.ACL 2025 · 10 citations
  4. Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze SurprisalSathvik Nair, Byung-Doh OhACL 2026 · 1 citation
  5. Are Language Models Any Good at Density Modeling?Sriram Ranga, Sai Shashank Bedampeta, Rui Mao, Anupam ChattopadhyayAAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines